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PREFACE 

The  Program  Evaluation  Committee  of  the  Massachusetts  Department  of  Education  has  prepared  this  guide  for 
school  districts  in  Massachusetts.  The  Program  Evaluation  Committee  is  an  ad-hoc  committee  of  the  Department 
made  up  of  personnel  from  all  major  program  units.  The  Committee  was  created  to  improve  program  evaluation 
activities  at  both  the  state  and  local  levels.  Preparing  this  guide  is  one  part  of  the  Committee's  effort  to  be  of 
assistance  to  local  school  districts. 

Several  school  districts  and  consulting  firms  provided  special  advice  and  assistance  in  the  process  of  producing 
this  guide.  They  took  a  special  interest  in  the  project.  Our  thanks  for  sharing  expertise  and  time  go  to: 

Ron  Chavious,  Evaluation  Specialist,  Boston  Public  Schools 

Robert  Consalvo,  Executive  Director,  Public  Affairs  Research  Institute 

Frank  DiGiammarino,  Lexington  Public  Schools 

Ronald  Nuttall,  Treasurer,  Vasquez-Nuttall  Associates 

Louis  Perullo,  Superintendent,  Whitman-Hanson  Regional  Schools 

John  Sullivan,  Grants  Manager,  Springfield  Public  Schools 

Joan  Wofford,  Leadership  and  Learning  Incorporated 

The  members  of  the  Program  Evaluation  Committee  hope  this  guide  will  be  useful  to  school  districts.  Please  feel 
free  to  contact  any  of  us  with  comments  or  suggestions  about  our  work.  We  are: 

Elizabeth  Badger,  Bureau  of  Research  and  Assessment 

Katherine  Best,  Northeast  Regional  Education  Center 

Monya  Geller,  Bureau  of  Nutrition  Education 

Carol  Gilbert,  Office  of  the  Deputy  Commissioner  for  Program  Operations 

Allan  Hartman,  Bureau  of  Research  and  Assessment 

Bambi  Levine,  Division  of  Special  Education 

Gerry  Mercadante,  Office  of  Executive  Planning 

Maxine  Minkoff,  Bureau  of  Educational  Resources 

Patricia  Mostue,  Division  of  Occupational  Education 

Marilyn  Pedalino,  Office  of  Executive  Planning 

Brunetta  Wolfman,  Office  of  Executive  Planning 


Digitized  by  the  Internet  Archive 

in  2013 


http://archive.org/details/guidetoevaluatioOOmass 


INTRODUCTION 

Program  evaluation  is  an  essential  part  of  every  school  manager's  job.  Its  intended  purpose  is  to  provide 
dependable  information  about  the  performance  of  educational  products,  practices  and  programs.  Decision 
makers  can  then  use  this  information,  along  with  other  data,  to  improve  the  quality  of  education  being  offered  in 
schools. 

The  Massachusetts  Department  of  Education  recognizes  the  importance  of  evaluation  as  part  of  the  school 
district's  own  program  planning  operation.  A  district  may  have  an  evaluation  specialist  or  other  staff  person  to 
conduct  most  of  the  evaluations  the  district  needs.  However,  sometimes  the  school  district  will  not  have  the 
in-house  expertise  to  conduct  the  evaluation  it  requires.  Other  times  the  school  district  will  determine  it  should  not 
conduct  an  evaluation  in-house. 

This  guide  has  been  written  to  assist  the  local  school  district  in  those  cases  where  an  outside  contractor  is  hired  to 
conduct  an  evaluation.  The  guide  is  written  specifically  for  the  evaluation  manager,  the  person  responsible  for 
having  the  evaluation  conducted.  It  presents  the  steps  a  manager  should  follow  to  assure  retrieving  the 
information  desired  from  the  evaluation.  It  is  not  a  guide  on  how  to  be  an  evaluator  and  conduct  the  evaluation.  Its 
focus  is  on  what  procedures  to  use  when  working  with  the  contractor.  Many  sample  documents  are  included  for 
references. 

Organization  of  the  Guide 

Section  I  presents  some  factors  to  consider  when  deciding  whether  to  conduct  an  evaluation  internally  or 
externally.  Section  II  describes  the  process  of  hiring  an  external  evaluator  for  a  large  contract,  $10,000  or  more, 
where  a  Request  for  Proposal  (RFP)  is  let  and  a  contract  is  awarded.  This  section  of  the  guide  goes  into  detail  on 
the  steps  of  writing  the  RFP,  selecting  the  evaluator,  negotiating  the  contract  and  monitoring  the  contract.  Section 
III  of  the  guide  describes  hiring  an  external  evaluator  for  a  small  contract,  under  $10,000.  This  section  describes 
topics  similar  to  those  in  Section  II,  but  gives  special  information  regarding  contracts  less  than  $2,000. 

The  definition  of  small  and  large  evaluations  is  a  general  guideline.  In  practice  a  district  may  select  amongst  the 
processes  discussed  under  each,  as  the  particular  evaluation  warrants. 

Evaluation  Standards 

Included  in  the  guide  for  general  reference  in  conducting  all  types  of  evaluation  studies  is  a  set  of  evaluation 
standards.  The  Massachusetts  Department  of  Education  has  adapted  these  standards  from  a  set  of  nationally 
developed  standards.  School  districts  are  encouraged  to  use  these  standards  whenever  possible  and  appropriate 
in  all  evaluation  efforts.  This  guide  will  not  examine  them  in  great  detail,  but  it  will  refer  to  them  in  the  text  of  the 
guide  when  they  apply.  The  standards  are  found  in  Appendix  A. 

This  guide  is  only  one  step  in  assisting  local  school  districts  to  improve  their  evaluation  efforts.  Many  other  issues 
in  evaluation  are  not  addressed.  We  welcome  the  comments  and  reactions  of  those  who  use  this  guide  with 
suggestions  for  additions,  deletions  or  revisions. 


I.  DECIDING  ON  AN  EXTERNAL  OR  INTERNAL  EVALUATION: 
THE  MANAGER'S  DECISION 

School  districts  conduct  evaluations  in  the  following  ways: 

Internally,  by  qualified  evaluation  specialists  in  the  district,  or  by  staff  of  the  program  being  evaluated. 

Externally,  by  independent  evaluators  chosen  by  the  school  district  on  the  basis  of  known  work  or  on  the  basis 
of  a  competitive  open  bid  process,  i.e.,  the  request  for  proposal  process.  An  external  evaluator  may  be  a  single 
individual  or  an  established  firm. 

When  the  program  evaluation  is  not  legally  mandated  from  funding  sources  to  be  either  internal  or  external,  a 
program  manager  must  make  that  decision.  Several  factors  should  be  taken  into  account: 

•  cost 

•  credibility  and  objectivity 

•  availability  of  qualified  personnel 

•  rapport  with  program  personnel 

Cost 

Using  an  outside  evaluator  requires  designating  budget  funds  which  sometimes  "seems"  more  expensive  than 
doing  an  evaluation  in-house  or  internally.  It  may  actually  be  more  expensive  if  there  are  unusually  high 
communication,  travel  or  overhead  costs.  Is  the  school  manager  able  to  justify  the  costs  and  gain  approval  for  this 
expenditure  in  time  for  the  evaluation  to  be  conducted  when  needed? 

Credibility  and  Objectivity 

Evaluations  need  to  be  conducted  so  that  the  results  are  believable  by  school  district  staff  and  outside  observers. 
When  an  evaluation  has  major  policy  implications  for  a  program,  credibility  and  objectivity  are  especially 
important.  An  external  evaluator  may  be  best  since  an  internal  evaluator  may  be  perceived  as  having  a  conflict  of 
interest.  The  external  evaluator  is  an  independent  agent.  However,  it  is  possible  that  in  a  particular  evaluation  the 
internal  evaluator  may  be  more  credible  than  available  external  evaluators. 

Availability  of  Qualified  Personnel 

If  a  school  district  does  not  have  a  qualified  evaluator,  then  in  the  ideal  case  the  district  must  hire  a  qualified 
external  evaluator.  Hiring  an  outside  evaluator  allows  the  district  to  have  access  to  persons  skilled  in  areas  beyond 
the  capacities  of  district  personnel.  It  can,  however,  be  difficult  to  determine  the  competence  of  an  outside 
evaluator  at  times,  particulary  if  the  evaluator  is  unknown  to  district  staff.  Even  if  a  school  district  has  a  qualified 
evaluator  in  the  district  (either  a  district-wide  evaluator  or  a  project  administrator  or  teacher),  this  person  may  or 
may  not  be  free  to  give  the  evaluation  its  due  time  and  energy  because  of  other  responsibilities.  Evaluations  are 
time  consuming.  The  evaluation  may  divert  the  project  person's  time  and  energy  when  it  is  more  important  for  the 
person  to  concentrate  on  the  day-to-day  operation  of  the  program. 

Rapport  with  Program  Personnel 

Evaluators  establish  a  working  relationship  with  program  personnel  in  the  course  of  conducting  the  evaluation.  If 
that  relationship  is  an  open  one,  with  frequent  communication  and  the  evaluator  is  familiar  with  the  project  being 
evaluated,  it  is  more  likely  that  the  evaluation  will  be  on  target  and  the  results  of  the  evaluation  will  be  implemented. 
For  a  particular  evaluation,  a  school  district  must  decide  how  important  this  rapport  is  for  the  success  of  the 
evaluation  and  whether  an  external  or  internal  evaluator  will  produce  the  best  results. 


II.     CONDUCTING  A  LARGE  EXTERNAL  EVALUATION  USING  A  REQUEST 

FOR  PROPOSALS  (RFP) 

For  a  large  evaluation,  $10,000  or  more,  a  school  district  would  most  likely  issue  an  RFP  to  solicit  bids  from 
prospective  contractors. 

Writing  and  Advertising  the  Request  for  Proposals 

General  Information 

An  RFP  is  a  notice  sent  to  prospective  contractors  informing  them  of  an  evaluation  that  a  district  intends  to 
sponsor.  The  purpose  of  issuing  an  RFP  is  to  identify  the  best  possible  approach  and  contractor  for 
conducting  the  evaluation.  This  is  usually  accomplished  by  a  competitive  bidding  process. 

The  local  school  district  issues  an  RFP  specifying  tasks  to  be  accomplished.  Contractors  then  submit 
proposals,  which  are  plans  for  how  they  would  do  the  work  outlined  in  the  RFP  and  what  they  would  charge. 
Then,  the  agency  reviews  the  proposals  and  selects  the  best  qualified  lowest  bidder  for  meeting  the  tasks  of 
the  particular  evaluation  project. 

Steps  in  Writing  an  RFP 

Clearly  state  the  purposes  of  the  evaluation,  the  type  of  evaluation  desired,  the  questions  to  be  answered, 

how  and  when  the  results  will  be  reported,  and  what  resources  are  available  to  conduct  the  evaluation. 

The  evaluation  manager  should  involve  other  people  in  a  discussion  of  these  issues  before  writing  an  RFP. 
After  agreement  has  been  reached  on  these  issues,  below  are  the  steps  to  follow  when  writing  the  RFP. 

1.  Answer  the  RFP  questions  carefully  and  completely  in  writing.  This  is  the  most  important  step  in  the  whole 
process  and  will  clarify  the  information  you  want  from  the  evaluation. 

What  programs  and/or  students  will  be  evaluated? 

What  questions  about  the  students  and/or  program  will  the  evaluation  address? 
What  instruments  or  alternative  methods  of  collecting  information  will  be  used? 
If  no  specific  instruments  or  methods  are  identified: 

What  general  types  of  instruments  will  be  selected? 

How  will  the  instruments  or  alternative  methods  be  selected  or  developed? 
What  are  the  qualifications  of  the  evaluator? 

If  the  evaluator  is  not  identified,  what  process  will  be  used  to  select  an  evaluator? 
What  are  the  dates  for  collecting  the  data  and  other  necessary  information  for  the  evaluation? 
When  will  interim  and  final  written  reports  be  submitted? 
How  much  money  or  resources  are  allocated  for  the  evaluation? 
How  will  the  results  of  this  evaluation  be  used? 
Who  will  receive  copies  of  the  evaluation? 

2.  Outline  the  RFP.  Develop  an  outline  using  the  sample  RFP  format  at  the  end  of  this  section.  Check  your 
outline  to  make  sure  that  there  is  a  place  to  include  all  of  the  information  from  above.  Basically,  your  outline 
will  start  with  more  general  information  about  the  project  or  program  to  be  evaluated.  Then,  it  will  be  moved 
toward  the  specifics  required  for  this  particular  evaluation. 

3.  Standard  information  that  is  basically  the  same  for  all  RFPs  is  provided  in  the  sample.  This  includes 
information  on  contracting,  submitting  bids,  printing  requirements,  and  the  proposal  review  process.  It  is 
unnecessary  to  change  these  sections  each  time  you  write  an  RFP.  Use  whatever  you  can  from  the  sample, 
making  only  those  changes  necessary  for  your  particular  situation. 

Be  sure,  as  part  of  the  proposal  review  section,  to  specify  the  components  of  the  proposal  to  be  reviewed  and 
the  number  of  points  to  be  awarded  for  each  component  (See  page  8).  Then,  both  proposal  reviewers  and 
potential  contractors  will  know  on  what  basis  they  will  be  judged. 

4.  Using  your  outline,  the  sample  RFP  format,  and  the  answers  to  the  questions  from  Step  1,  you  are  ready  to 
begin  writing  the  RFP.  The  way  to  do  this  is  to  transfer  the  answers  to  the  RFP  Questions  into  thebegin  writing 


the  RFP.  The  way  to  do  this  is  to  transfer  the  answers  to  the  RFP  Questions  into  the  appropriate  sections  of 
the  sample  format. 

5.  After  the  RFP  is  drafted,  do  a  rough  budget  estimate  for  the  work  you  have  described.  In  order  to  do  this 
estimate,  you  could  check  other  budgets  for  similar  evaluations  already  completed  from  persons  in  or  out  of 
your  school  district  as  well  as  your  school  district  budget  personnel.  Include  major  costs,  such  as  evaluator 
time  and  large  numbers  of  printed  materials.  The  purpose  of  this  is  not  to  develop  a  budget  that  will  actually  be 
used.  The  purpose  is  to  check  to  make  sure  that  the  tasks  required  can  reasonably  be  done  for  the  amount  of 
money  available.  If  your  cost  estimates  are  much  higher  than  the  money  available,  you  may  need  to  rethink 
what  you  are  requesting  in  the  RFP. 

6.  After  a  draft  of  the  RFP  is  written,  it  is  helpful  to  circulate  it  to  others  for  their  reactions  and  comments.  Other 
people  can  often  help  point  out  when  a  section  is  unclear,  or  when  additional  information  is  needed.  At  the 
very  least,  other  staff  members  involved  in  the  project  should  review  the  RFP.  You  may  also  want  to  have  a 
technical  consultant  who  will  not  be  a  bidder  review  the  RFP,  particularly  if  there  are  complex  technical  issues. 
Depending  upon  the  amount  of  time  available,  you  may  also  wish  to  have  the  draft  RFP  reviewed  by  members 
of  any  panel  who  will  assist  you  in  screening  proposals.  (See  Political  Viability  Standard  page  40). 

7.  After  reviewing  all  of  these  comments,  make  any  necessary  revisions  in  your  draft.  The  RFP  is  now  complete. 

Do's  and  Don'tsfor  Writing  an  RFP 

DO  issue  the  RFP  with  enough  lead  time  so  that  prospective  contractors  can  do  a  good  job  with  the  proposal. 

Do  be  precise  about  the  end  results  of  the  evaluation.  State  clearly  what  you  want  to  know,  how  precise  the  results 
must  be,  and  how  the  results  are  to  be  reported. 

DO  indicate  the  amount  of  money  available  for  conducting  the  evaluation. 

DO  list  clearly  the  products  that  will  result  from  the  contract.  Include  a  description  of  each  item,  number  of  copies 
required,  and  date  required. 

DO  specify  any  dates  for  which  the  contractor's  presence  is  required  (such  as  school  committee  meetings),  as  well 
as  a  list  of  crucial  dates  for  the  project. 

DO  specify  clearly  who  has  responsibility  (contractor  or  district)  for  costs  of  mailing,  duplicating  materials, 
contracting  participants,  paying  for  travel  and  meals  for  participants,  etc. 

DO  provide  complete  information  on  what  criteria  will  be  used  when  you  review  the  RFP  and  how  the  review 
process  will  be  conducted. 

DO  include  background  information  on  the  project. 

DO  specify  any  format  and/or  content  requirements  for  the  budget. 

DO  include  an  evaluation  design,  if  one  has  already  been  selected. 

DON'T  provide  details  on  how  the  contractor  should  go  about  accomplishing  the  task;  that  is  the  job  of  the  bidder. 
Different  contractors  will  propose  different  ways  to  get  the  job  done.  As  long  as  you  have  been  clear  about  what 
the  job  is,  you  will  be  able  to  compare  the  approaches  and  costs  proposed  by  different  bidders  and  select  the  one 
that  you  think  is  the  best. 

DONT  be  too  general  and  ambiguous. 

DONT  omit  key  issues  you  want  to  see  addressed  in  the  proposals. 

DONT  use  jargon. 

DONT  expect  a  large  response  for  a  small  RFP. 

DONT  establish  unrealistic  time  or  budgetary  constraints  for  the  tasks  required. 


REMEMBER:  IT  IS  IN  YOUR  BEST  INTEREST  FOR  BIDDERS  TO  KNOW  WHAT  YOU  NEED.  TELL 
CONTRACTORS  WHAT  YOU  WANT  AND  YOU  STAND  A  MUCH  BETTER  CHANCE  OF  GETTING  IT. 

Sample  RFP  Form 

A  sample  form  is  provided  at  the  end  of  this  Section. 

The  nature  of  the  task,  the  amount  of  money  available,  and  the  complexity  of  what  is  required  will  affect  how  the 

RFP  should  be  written.  This  is  only  one  suggested  format.  Other  formats  may  work  equally  well,  as  long  as  all 

necessary  information  is  provided.  If  you  have  never  written  an  RFP  before,  you  may  want  to  start  out  with  this 

format  and  modify  it  as  you  go  along.  Once  you  have  developed  your  own  format,  you  can  use  that  as  a  guideline  in 

developing  future  RFPs. 

Ways  to  Advertise 

There  are  two  ways  to  advertise  an  RFP.  You  may  wish  to  use  one  or  both  of  these  methods.  In  any  case,  an  RFP  is 

public  information  and  must  be  distributed  widely. 

First,  you  may  announce  the  availability  of  an  RFP  through  the  mail  (to  a  list  of  potential  bidders)  or  through  a 
newspaper  advertisement.  The  announcement  should  include,  at  a  minimum,  the  following  information: 

•  A  heading  titled  "Request  for  Proposals". 

•  The  name  of  the  unit  in  the  school  district  issuing  the  RFP. 

•  The  intent  of  the  request,  the  desired  program  and/or  products  to  be  evaluated  and  intended  recipients  of  the 
evaluation. 

•  Identification  of  a  contact  person  with  directions  for  obtaining  further  information. 

•  The  amount  of  money  available. 

•  The  deadline  date  for  submission  of  final  proposals. 

Second,  you  may  identify  a  list  of  potential  bidders  and  send  the  RFP  out  directly.  If  you  do  not  send  out  an 
announcement,  make  sure  that  you  send  the  RFP  to  a  reasonable  number  of  potential  bidders.  Be  sure  to  give  new 
firms  and  minority  firms  a  fair  chance  to  bid. 

Since  mailing  and  printing  costs  should  be  kept  to  a  minimum,  an  announcement  of  the  RFP  to  potential  bidders  is 
recommended  prior  to  sending  out  the  complete  RFP.  Then,  after  a  written  Intent  to  Bid  is  received, 
accompanying  materials  can  be  duplicated  and  mailed. 

Document  all  procedures  followed  in  advertising  the  RFP.  This  information  will  be  needed  to  develop  and  process 
the  resulting  contract. 

Public  Information  Requirements 

Massachusetts  has  a  far-reaching  public  records  law.  Anyone  has  the  right  to  see  and  have  a  copy  of  almost  any 
written  document  made  or  received  by  any  public  agency,  including  a  school  district.  To  comply  with  the  public 
records  law  and  to  be  fair  to  all  potential  bidders,  you  should  note  the  following: 

An  RFP  is  public  information.  Make  the  RFP  available  to  anyone  who  specifically  requests  a  copy. 

Maintain  documentation  of  any  advertisement  and  bidders  list  used  for  distributing  the  RFP. 

The  RFP  should  include  clear  deadlines  for  submission  of  bids  and  written  intention  to  bid,  as  well  as  any  dates 
after  which  requests  for  information  will  not  be  honored.  It  is  essential  that  you  strictly  observe  these  deadlines.  If 
you  change  any  deadlines,  make  sure  everyone  gets  the  new  information  at  the  same  time. 

If  bidders  are  required  to  submit  a  written  Intent  to  Bid,  the  names  of  those  submitting  such  letters  are  public 
information. 

Make  yourself  equally  available  to  all  potential  bidders  for  answering  questions  and  providing  information.  Any  of 
the  methods  stated  below  are  fine  as  long  as  the  method  is  stated  in  the  RFP  itself. 


Selecting  the  Evaluator 

Proposal  Review  Process 

(See  Evaluator  Credibility  Standard  page  39  and  Conflict  of  Interest  Standard  page  41.) 

General  Information 

Procedures  for  reviewing  proposals  and  selecting  an  evaluator  should  be  established  and  included  in  the  RFP.  The 

review  procedures  should  include  information  on: 

•  What  groups  or  committees  will  review  proposals. 

•  The  elements  that  will  be  considered  in  reviewing  proposals,  including  the  maximum  possible  points  that  a 
proposal  may  receive  for  each  element. 

•  You  can  hold  a  public  bidders'  meeting  at  which  time  questions  about  the  RFP  can  be  answered.  Such  a 
meeting  should  be  publicly  advertised  and  open  to  all  interested  parties. 

•  You  can  also  be  available  to  answer  telephone  inquiries  about  the  RFP.  In  this  case,  you  should  keep  a  log  of  all 
telephone  calls.  Items  to  be  recorded  in  a  log  include:  the  name  and  firm  of  a  person  requesting  information; 
address,  date,  and  nature  of  the  request;  and  the  date  and  nature  of  the  response.  This  log  must  be  available  for 
public  inspection. 

•  If  telephone  calls  will  not  be  accepted,  the  RFP  should  state  that  interested  applicants  seeking  further 
information  should  submit  inquiries  in  writing.  All  written  requests  for  information  and  written  responses  are 
public  information. 

Until  the  deadline  for  submission  of  proposals,  the  bids  (proposals)  submitted  are  not  public  information. 
However,  once  the  bid  deadline  is  passed,  all  bids  received  become  public  information,  even  if  the  proposal 
contains  a  statement  to  the  contrary. 

Materials  used  in  the  review  of  proposals  should  be  kept  confidential  during  the  review  process.  However,  once  a 
decision  has  been  made  to  enter  into  negotiations  with  a  bidder  or  award  a  contract,  all  records  of  the  review 
process  may  no  longer  be  kept  confidential;  they  become  public  information. 

You  can  require  that  someone  requesting  a  copy  of  a  public  document  pay  for  the  cost  of  duplicating  the  material. 
The  person  making  the  request  can  send  you  a  check  made  out  directly  to  a  photocopy  place.  You  can  then  have 
the  copy  made  and  pass  on  the  check  directly  to  the  photocopier. 

If  you  wish  to  have  the  person  requesting  the  material  pay  mailing  costs,  have  them  send  you  a  stamped, 
self-addressed  envelope  (specify  size  of  envelope  and  amount  of  postage  required). 

The  pertinent  section  of  the  Public  Records  Law  is  cited  here  from  the  General  Laws  Chapter  4, 
Section  7(26): 

"Public  Records"-Twenty-sixth,  "Public  records"  shall  mean  all  books,  papers,  maps,  photographs,  recorded 
tapes,  financial  statements,  statistical  tabulations,  or  other  documentary  materials  or  data,  regardless  of  physical 
form  or  characteristics,  made  or  received  by  any  officer  or  employee  of  any  agency,  executive  office,  department 
board,  commission,  bureau,  division  or  authority  of  the  Commonwealth,  or  of  any  political  subdivision  thereof,  or 
of  any  authority  established  by  the  general  court  to  serve  a  public  purpose,  unless  such  materials  or  data  fall  within 
the  following  exemptions  in  that  they  are: 

(h)  proposals  and  bids  to  enter  into  any  contract  or  agreement  until  the  time  for  the  opening  of  bids  in  the 
case  of  proposals  or  bids  to  be  opened  publicly,  and  until  the  time  for  the  receipt  of  bids  or  proposals  has 
expired  in  all  other  cases;  and  inter-agency  or  intra-agency  communications  made  in  connection  with  an 
evaluation  process  for  reviewing  bids  or  proposals,  prior  to  a  decision  to  enter  into  negotiations  with  or  to 
award  a  contract  to,  a  particular  person. 


It  is  unlikely  that  a  proposal  will  be  outstanding  or  seriously  deficient  in  everything.  The  review  process  should 
highlight  the  relative  merits  and  problems  of  each  proposal. 

Points  can  be  awarded  for  such  categories  as: 

•  Quality  of  products  and  activities. 

•  Costs  in  relation  to  products. 

•  Thoroughness,  adequacy,  and  clarity  of  budget. 

•  Responsiveness  to  the  questions  posed  in  the  RFP. 

•  Quality  of  the  management  plan. 

•  Qualifications  and  time  allocations  of  project  staff. 

•  Contractor's  resources  and  relevant  experience  in  evaluation. 

•  Creativity  of  the  proposal. 

These  are  only  examples.  Depending  upon  the  purposes  of  the  RFP,  you  may  wish  to  add  or  delete  certain 
categories,  separate  some  categories  into  two,  or  combine  some  categories. 

The  way  points  are  distributed  depends  upon  the  purposes  of  the  study  and  how  you  choose  to  weigh  the  different 
components  of  proposals.  If,  for  instance,  the  project  involves  test  development  and  data  analysis,  you  probably 
would  award  fewer  points  for  creativity,  since  creativity  is  not  relevant.  In  fact,  you  may  eliminate  the  whole 
category.  If  you  will  be  looking  for  specific  characteristics  (such  as  evidence  of  effective  work  with  advisory 
groups),  then  you  should  specify  that  and  designate  a  certain  number  of  points  for  that  category. 

It  is  important  to  specify  ahead  of  time  (for  inclusion  in  the  RFP)  the  components  of  the  proposal  to  be  reviewed 
and  the  number  of  points  to  be  awarded  for  each  component.  Then,  both  proposal  reviewers  and  potential 
contractors  will  know  on  what  basis  they  will  be  judged. 

Steps  in  Proposal  Review  Process 

Once  the  methods  and  content  of  proposal  review  have  been  established,  follow  these  steps: 

1 .  Organize  an  ad  hoc  committee  to  review  proposals.  This  committee  should  always  include  staff  from  the  unit 
that  sent  out  the  RFP.  You  may  also  want  to  include  representatives  from  other  units  and  state  or  local 
associations  who  would  have  an  interest  in  the  evaluation  findings.  The  size  of  a  review  committee  may  range 
from  about  6  to  12  people,  although  this  number  is  only  a  guideline.  If  non-district  reviewers  are  sitting  on  the 
committee,  it  should  be  made  clear  that  their  recommendations  will  be  taken  seriously,  but  that  the  final 
decision  must  be  made  by  district  staff  and/or  the  school  committee. 

2.  Get  technical  assistance  if  necessary.  In  addition  to  a  review  by  district  staff  and  other  interested  parties,  you 
may  also  need  outside  technical  assistance.  This  would  be  appropriate  if  the  size  of  the  contract  is  large,  if 
there  are  technical  issues  unfamiliar  to  district  staff,  or  if  political  challenges  to  the  evaluation  are  likely.  The 
task  of  the  technical  expert  is  to  review  proposals  and  write  up  a  critique  of  each  proposal.  These  critiques 
should  then  be  shared  with  district  staff  and  with  the  various  contractors  and  help  school  district  staff  identify 
necessary  amendments  to  the  winning  proposal. 

Technical  experts  can  be  people  who  have  the  necessary  knowledge  of  the  subject  and  of  the  type  of 
evaluation.  (For  instance,  if  a  complex  sampling  plan  is  to  be  used,  you  would  want  to  find  someone 
knowledgeable  about  sampling.)  Make  sure  that  the  skills  of  your  technical  expert  are  appropriate  to  the 
specific  task.  If  outside  technical  consultants  are  used  to  advise  on  the  quality  of  proposals,  make  certain  that 
no  conflict  of  interest  exists.  Obviously,  someone  serving  as  a  technical  consultant  could  not  also  be  a  bidder. 
Nor  should  a  technical  consultant  be  associated  in  any  way  with  a  bidding  consultant  firm. 

(See  Practical  Procedures  Standard  page  40  for  further  considerations.) 

3.  Arrange  for  a  time  and  location  for  a  meeting  to  review  the  proposals  and  interview  finalists.  Send  out 
necessary  materials  to  the  review  committee.  Committee  members  should  receive: 

•     RFP  and  accompanying  materials  sent  to  potential  bidders. 


•  Proposals.  (Make  sure  you  require  bidders  to  submit  enough  copies  for  all  members  of  the  review 
committee,  plus  what  you  will  need  for  processing  and  monitoring  the  contract.) 

•  Directions  for  reviewing  proposals. 

•  Rating  forms  (one  for  each  proposal). 
Any  critiques  or  comments  written  by  technical  consultants. 


• 


4.  Narrow  down  prospective  bidders.  When  proposals  are  received,  staff  should  quickly  review  them  (using 
criteria  specified  in  the  RFP)  to  see  if  any  are  extremely  weak  and  should  not  be  pursued.  This  decision  may  be 
made  by  school  district  staff,  or  you  may  wish  to  poll  members  of  the  review  committee  by  telephone  to  get 
their  recommendations  about  which  proposals  merit  further  consideration.  Once  you  determine  which 
bidders  to  interview,  you  can  schedule  the  interviews  with  bidders. 

5.  Plan  the  agenda  for  the  proposal  review  meeting.  Contractors  should  send  one  or  two  key  people  to  the 
interview.  The  same  amount  of  time  should  be  allocated  for  each  interview.  The  amount  of  time  allocated  will 
depend  upon  the  number  of  bidders  you  wish  to  interview  and  the  length  of  time  the  review  committee  will 
meet.  Generally,  an  hour  or  more  should  be  scheduled  for  each  interview. 

Prior  to  the  first  interview,  schedule  an  hour  or  so  for  review  panel  members  and  technical  consultants  (if  any) 
to  discuss  the  types  of  questions  that  need  to  be  raised  with  the  bidders. 

6.  Interview  bidders.  The  evaluation  manager  for  the  contracting  unit  should  chair  this  session.  Careful  notes 
should  be  kept  during  the  interviews.  Since  the  bidders  know  they  are  in  final  competition  for  the  contract  at 
the  time  of  the  interview,  you  are  in  a  good  position  to  gain  favorable  clarifications  of  anything  that  is  unclear  in 
the  proposal.  Bidders  should  be  told  that  they  will  be  held  to  anything  they  say  during  the  interview.  Verbal 
agreements  should  then  be  written  as  an  amendment  to  the  proposal  of  the  successful  bidder  (See  page  9 
Negotiating  a  Contract). 

After  all  contractors  have  been  interviewed,  review  panel  members  should  discuss  their  reactions,  focusing  on 
perceived  strengths  and  weaknesses  of  each  proposal.  After  this  discussion,  panel  members  should  revise  the 
review  sheets  completed  prior  to  the  interview  on  the  basis  of  what  they  learned  from  the  session.  Each 
member  should  then  make  a  recommendation  and  give  reasons  for  the  selection. 

7.  Select  a  proposed  contractor.  Based  on  the  recommendations  of  outside  reviewers,  technical  consultants, 
and  staff,  a  winning  proposal  should  be  tentatively  identified. 

8.  Notify  successful  and  unsuccessful  bidders. 

Sample  Proposal  Review  Sheet 

An  example  of  a  proposal  review  sheet  is  attached  here.  It  lists  the  different  elements  of  the  proposal  which  will  be 
considered  in  its  review.  In  addition,  the  maximum  number  of  points  that  may  be  awarded  is  specified  for  each 
element.  Reviewers  have  a  place  to  indicate  their  initial  score  (prior  to  interviewing  the  bidder),  as  well  as  their  final 
score  (following  the  interview  and/or  discussion  with  other  committee  members). 

Remember  that  this  is  only  an  example.  Each  RFP  needs  to  specify  the  elements  that  will  be  considered  in 
reviewing  proposals,  as  well  as  the  point  distribution.  This  will  vary  from  RFP  to  RFP.  In  setting  up  the  review 
process,  you  need  to  identify  those  elements  of  proposals  that  will  be  of  most  importance  for  each  situation. 

Make  sure  to  collect  all  review  forms  and  keep  them  as  part  of  your  documentation  of  the  selection  process. 
Losing  bidders  may  ask  to  see  the  review  forms  to  improve  their  chances  of  being  awarded  a  contract  in  the 
future. 
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SAMPLE  PROPOSAL  REVIEW  SHEET 


RFP  Title: 


Bidder: 


RFP#: 


DIRECTIONS  TO  REVIEWERS:  Reread  the  RFP,  and  note  the  elements  of  the  proposal  for  which  points  will 
be  awarded  (see  below).  Then,  carefully  review  each  proposal  and  indicate  your  initial  point  allocation. 
Following  the  interview  of  bidders  and/or  discussion  with  other  reviewers,  indicate  your  final  ratings  for 
each  proposal.  Add  any  comments  at  the  bottom  of  the  page.  Complete  one  Review  Sheet  for  each  proposal 
that  you  are  reviewing.  Ratings  will  be  available  for  public  review  by  contractors  and  other  interested  parties 
upon  request. 


Points  Awarded 


Element  of  Proposal 


Possible 
Points 


Initial 
Review 


Final 
Review 


1.  Objectives — Are  they  clearly  stated 
and  appropriate  for  the  RFP? 

15 

2.  Activities — Are  they  related  to  the 
objectives  and  fulfill  all  requirements 
of  the  RFP? 

25 

3.  Qualifications  of  proposed  staff — Do 
they  have  the  training  and  experience 
pertinent  to  the  project? 

25 

4.  Management  Plan — Is  the  project-time  plan 
clearly  presented  with  appropriate  personnel 
allocated  to  each  activity? 

20 

5.  Budget — Is  the  budget  clear  and  realistic? 

15 

Total 


100 


Comments: 


< 
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Negotiating  a  Contract 

Certain  kinds  of  negotiations  are  acceptable,  and  others  are  not.  Changes  should  not  be  made  in  the  scope  of 
work  or  outcomes  described  in  the  RFP.  However,  it  is  acceptable  to  negotiate  changes  with  a  bidder  about  how 
the  work  tasks  will  be  carried  out.  During  the  interview,  bidders  will  usually  indicate  the  extent  of  changes  they  are 
willing  to  make  in  their  proposal. 

After  the  initial  identification  of  the  successful  bidder,  the  review  panel  and  technical  experts  should  turn  their 
attention  to  necessary  modifications  to  the  selected  proposal.  For  example,  the  review  panel  might  suggest 
changes  in  the  sampling  procedures,  hypotheses  to  be  tested,  or  budget  allocations.  Reviewers  should  keep 
budgetary  constraints  in  mind  when  requesting  amendments.  Again,  very  careful  minutes  of  changes  requested 
should  be  kept. 

Based  on  the  notes  of  the  review  committee  meeting,  as  well  as  a  review  of  the  tasks  required  by  the  RFP,  the 
school  district  evaluation  manager  should  write  a  formal  Request  for  Amendment  to  the  winning  bidder.  This  can 
be  in  the  form  of  a  letter  or  memorandum.  In  addition,  the  evaluation  manager  will  probably  want  to  meet  with  the 
bidder  to  review  the  amendments  required.  The  bidder  then  submits  a  memorandum  amending  the  original 
proposal.  Both  the  Request  for  Amendment  and  the  amendment  memorandum  become  part  of  the  formal 
contractual  obligation  of  the  bidder. 

Remember  that  a  Contract  Management  Plan  will  be  written  to  detail  and  refine  the  proposal  work  (see  page  11). 
Be  sure  proposal  changes  are  actually  new  or  different  items. 

When  you  notify  the  winning  bidder,  the  notice  must  make  it  clear  that  this  selection  does  not  constitute  formal 
approval  of  the  contract.  No  individual  employee  of  a  district  may  formally  award  a  contract  or  authorize  work  to 
begin  without  official  school  committee  approval.  It  is  especially  important,  therefore,  that  adequate  time  is 
allowed  for  this  final  approval,  so  that  the  contract  work  can  proceed  on  schedule. 

Monitoring  and  Evaluating  the  Evaluation  Contract 

Monitoring  Procedures 

Monitoring  an  evaluation  contract  is  an  essential  part  of  the  evaluation  manager's  job.  No  outside  evaluator  should 
be  hired  to  complete  the  job  without  careful  monitoring  by  the  evaluation  manager.  Monitoring  does  not  mean 
getting  involved  with  the  details  and  day-to-day  activities  of  carrying  out  the  evaluation.  However,  the  evaluation 
manager  must  keep  track  of  progress,  approve  of  key  decisions,  and  make  sure  that  crucial  deadlines  are  met. 

There  are  four  vehicles  for  effective  monitoring  of  an  evaluation  contract. 

a.  The  best  monitoring  tool  is  a  good  contract  (including  the  RFP,  Proposal,  Request  for  Amendment  Letter  and 
Proposal  Amendment),  and  the  most  important  part  for  monitoring  is  a  good  RFP.  If  the  RFP  is  clear  and 
specific  about  the  tasks  to  be  accomplished,  the  amount  of  money  available,  the  materials  to  be  produced, 
how  materials  will  be  distributed,  meeting  dates  for  which  the  evaluator's  presence  is  required,  how  money  will 
be  spent  and  reallocated  if  necessary,  and  specific  dates  for  completion  of  major  project  tasks,  half  the  work  of 
monitoring  a  contract  is  done.  What  the  evaluation  manager  needs  to  do  is  to  refer  frequently  to  the  RFP  to 
ensure  that  tasks  are  being  accomplished  as  specified. 

b.  After  a  contract  has  been  awarded,  the  first  thing  the  contractor  should  do  is  prepare  a  detailed  Contract 
Management  Plan.  (See  sample  page  11).  The  plan  should  list  specific  dates,  tasks,  and  responsibilities  for 
accomplishing  the  tasks  required  by  the  project  and  outlined  in  the  RFP  and  proposal.  It  is  a  sequential, 
operational  plan  that  grows  out  of  the  RFP  and  the  proposal,  as  well  as  any  amendments.  The  Plan  should  be 
submitted  in  draft  form  to  the  evaluation  manager,  and  may  also  be  reviewed  by  appropriate  advisory 
committees.  It  is  essential  that  the  plan  be  reviewed  very  carefully  for  completeness  and  accuracy.  Time  and 
care  spent  at  this  crucial  stage  will  make  monitoring  of  the  contract  much  easier  during  the  entire  contract 
period. 


After  the  evaluation  manager  has  scrutinized  the  Contract  Management  Plan  and  gotten  comments  from 
others,  changes  should  be  given  to  the  contractor,  who  should  then  submit  a  revised  management  plan 
document.  This  plan  becomes  the  schedule  for  carrying  out  the  evaluation.  The  evaluation  manager  can  then 
simply  check  to  make  sure  that  necessary  tasks  are  being  accomplished  by  the  specified  dates. 

c.  No  matter  how  carefully  the  RFP  and  Contract  Management  Plan  are  developed,  reviewed,  and  revised,  there 
will  almost  always  be  some  changes  along  the  way.  All  prepared  changes  should  be  reviewed,  agreed  upon  by 
both  parties  and  documented. 

The  contractor  should  send  memos  confirming  any  agreed  upon  changes.  Changes  could  be  in  the  specific 
way  tasks  will  be  carried  out,  trade-offs  (more  copies  of  a  report,  but  fewer  pages  than  planned),  types  of 
analysis  to  be  conducted,  etc.  It  is  especially  important  to  document  any  changes  which  affect  the  budget.  The 
evaluation  manager  should  carefully  review  memos  on  changes  and  make  necessary  corrections  immediately. 

d.  In  general,  school  district  staff  should  review  all  draft  materials  produced  by  the  contractor  before  printing  and 
distribution.  The  review  should  be  limited  to  factual  information,  completeness  or  clarity.  Materials  included 
in  this  category  are  :  questionnaires;  letters  to  parents,  teachers  or  students;  workshop  materials;  final 
reports;  any  other  printed  materials.  Remember  it  is  unethical  to  ask  a  contractor  to  change  results  or  to  omit 
important  findings. 

In  rare  instances,  an  evaluator  is  hired  to  make  an  independent  judgment  about  an  issue  and  submit  a  report 
to  the  district.  In  these  cases,  the  contractor  would  retain  complete  control  over  the  contents  of  the  report 
and  the  district  would  not  review  materials.  The  RFP  would  state  who  has  control  over  the  content  of  the 
report. 

Prior  to  processing  final  payment,  the  evaluation  manager  should  carefully  review  all  project  documents  (RFP, 
Proposal,  Amendment  Request,  Proposal  Amendment,  documentation  memos).  Final  payment  should  not  be 
processed  until  all  materials  are  received  and  all  tasks  completed  as  specified  in  the  project  documents. 

The  Contract  Management  Plan 

On  the  following  page  is  an  excerpt  from  a  Contract  Management  Plan  for  a  program  evaluation.  Key  dates  are 
specified,  along  with  essential  evaluation  project  activities.  In  addition,  those  responsible  for  each  activity  are 
identified. 

Certain  activities  in  a  program  evaluation  are  ongoing,  with  a  deadline  date  indicated.  Others  require  meetings  or 
telephone  calls  on  specified  dates. 

The  Plan  is  of  mutual  benefit.  It  spells  out  the  obligations  of  both  the  contractor  and  the  evaluation  manager 
responsible  for  monitoring  the  contract.  You  are  obligated  to  meet  your  responsibilities  as  much  as  the  contractor 
is  obligated  to  meet  his/her  responsibilities. 

Evaluating  the  Contract 

A  school  district  should  develop  a  plan  to  evaluate  the  quality  of  evaluation  contract  work.  This  activity  is 
important  for  a  number  of  reasons: 

•  school  districts  must  have  some  way  of  knowing  the  degree  to  which  evaluation  results  are  valid  and  reliable, 
and  useful  for  making  decisions  about  the  future  of  programs  evaluated; 

•  school  districts  should  have  a  formal  process  for  insuring  that  all  work  contracted  for  has  been  completed  in 
the  expected  fashion; 

•  evaluation  contractors  should  be  given  formal  feedback  on  the  degree  to  which  the  work  which  they  have 
submitted  has  met  the  district's  expectations,  and  subsequently  the  possibility  of  future  contracts. 

There  are  several  options  for  accomplishing  this  task  including  independent  review  panels  of  evaluation  "experts" 
and  public  hearings  of  the  findings.  A  district  must  weigh,  however,  the  costs  both  in  dollars  and  personnel  time 
involved  to  determine  how  elaborate  a  review  is  necessary.  In  many  cases  simply  providing  the  mechanism  for  the 
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A  SAMPLE  CONTRACT  MANAGEMENT  PLAN 


UU 

< 


UJ 

Cu 

< 


Cu 

< 

UJ 

< 

UJ 


Date 


Activity 


By  Whom 


August  25 


Orientation  meeting  (1/2  day) 


September  1 

Submit  preliminary  evaluation  d 

September  6 

Administer  reading  test 

September  12 

Submit  list  of  target  population 
with  names  and  addresses  of 
parents/guardians 

September  20 

Submit  list  of  selected  sample 

September  25 

Meeting  to  review  evaluation 

design;  assessment  instruments, 
etc.  (V2  day) 

October  15 

Administer  achievement  tests 

October  16-30 

Analyze  achievment  data 

February  1 

Meeting  to  review  logs, 
observation  and  achievement  di 

February  15 

Submit  Interim  report 

May  20 

Questionnaire  ready  for  review 

June  1 

Administer  achievement  tests 

June  15 

Administer  questionnaire  to 
Title  VII  principals 

July  1-15 

Analyze  data 

July  30 

Submit  final  report 

Evaluator,  Superintendent, 
Project  Staff 

Evaluator 

Schools 

Project  Staff 

Evaluator 

Evaluator,  Superin- 
tendent, Project  Staff 
Liaison  Worker 

Project  Staff 

Evaluator 

Project  Staff, 
Liaison  Worker 

Evaluator 

Evaluator 

Project  Staff 

Evaluator 
Evaluator 
Evaluator 


Meeting  between  Evaluator  and  Project  Staff  will  take  place  on  the  2nd  Tuesday  of  each  month         w 
(October-June)  unless  another  day  is  mutually  agreed  upon.  iii 
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review  in  a  letter  or  meeting  will  be  enough  to  provide  formal  feedback  to  the  contractor.  The  key  elements  in  this 
feedback  are: 

•  comparison  of  work  expected  or  written  in  the  contract  compared  with  work  received; 

•  comparison  of  contractor's  work  with  appropriate  Program  Evaluation  Standards,  as  developed  by  the 
Department  of  Education  (Appendix  A),  which  the  contractor  and  school  district  have  previously  agreed  upon 
as  important. 

Non-Compliance  Procedures 

There  are  various  levels  at  which  a  contract  can  be  judged  to  be  unsatisfactory.  In  general,  the  range  of  recourses 
available  is  related  to  the  extent  and  type  of  non-compliance.  When  an  evaluation  service  that  is  specifically 
contracted  for  is  not  delivered,  the  school  district  may  withhold  payment  and  instigate  legal  action.  On  the  other 
hand,  if  the  evaluation  is  judged  inadequate  in  terms  of  quality,  there  are  fewer  options  for  recourse.  Even  when 
qualitative  criteria  have  been  specified  in  the  contract,  it  may  be  difficult  to  establish  inadequacies  because  of  the 
subjective  nature  of  qualitative  judgments.  In  these  cases,  an  effective  course  of  action  is  relying  upon  the 
contractor's  value  of  a  good  reputation  and  opportunity  for  future  employment.  Below  are  various  strategies, 
listed  in  terms  of  severity.  These  steps  may  also  be  regarded  as  sequential,  i.e.,  a  series  of  actions  that  may  be 
followed  to  final  resolution  of  the  case. 

1.  Informal  letter  of  inquiry. 

As  a  first  step,  an  informal  letter  may  be  sent  to  the  contractor  requesting  further  material,  refinement, 
clarification,  modification,  etc.  This  will  allow  the  contractor  to  explain  circumstances  that  may  relate  to 
performance  and  will  afford  the  opportunity  to  remedy  defects. 

2.  Formal  meeting  of  all  parties. 

A  meeting  with  the  head  of  the  contractor's  company,  his  staff  working  on  the  evaluation  and  school  district 
staff  could  be  called  to  resolve  the  problems. 

3.  Formal  letter  of  complaint  with  a  copy  to  the  Department. 

This  letter  can  serve  two  purposes.  It  can  formally  detail  the  inadequacies  in  the  evaluation,  in  preparation  for 
future  legal  action.  It  can  also  help  to  ensure  that  the  relative  reliability  of  a  contractor  is  more  generally 
known.  Contractors  who  value  their  good  name  are  likely  to  rectify  inadequacies  without  further  action  on  the 
part  of  the  district.  It  is  important  for  the  educational  community  to  be  aware  of  unsatisfactory  performance 
on  the  part  of  the  contractor.  Any  allegations  of  incompetence  or  wrong  doing  must  be  carefully  documented. 

4.  Negotiated  settlements  of  payment. 

A  contract  is  not  one-sided;  both  parties  must  live  up  to  their  contractual  obligations.  For  this  reason, 
contracts  should  specify  that  final  payment  be  withheld  until  the  satisfactory  completion  of  the  proposed 
work.  It  is  also  useful  to  include  within  the  contract  a  stipulation  that  it  may  be  terminated  for  non- 
performance or  inadequate  performance  at  any  time.  In  the  event  of  such  termination,  it  is  noted  that 
compensation  be  adjusted  to  the  percentage  of  performance  completed. 

Before  deciding  on  such  a  course  of  action,  legal  advice  should  be  sought.  The  strength  of  the  school  district's 
case  is  contingent  upon  how  well  it  can  document  incompetency.  Ideally,  the  contract  should  include  an 
explicit  criteria  for  performance.  In  this  case,  non-compliance  is  most  easily  documented.  However, 
inadequacies  in  terms  of  unspecified  criteria  can  be  challenged  provided  the  work  has  not  been  accepted 
previously.  Since  acceptance  implies  tacit  approval,  it  follows  that  contracts  should  be  monitored  carefully, 
the  contractor  should  be  made  aware  of  expectations,  and  objections  should  be  made  promptly  when 
inadequate  performance  occurs. 

(See  Accuracy  Standards  page  42  and  Formal  Obligation  Standard  page  41  for  additional  information) 
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Sample  RFP  for  a  Large  Evaluation  ^ 

< 

REQUEST  FOR  PROPOSALS  <*> 


uu 


Title:     Evaluation  of  the  Pre-School  Language  Development  Program ■■ 

Date  Issued:     May  15. 1982 5 

< 

</5 


For:     School  Board  of  Cosmopolis 
Bureau/Division/Office:     Division  of  Research 


Address:     108  Main  Street 


Available  Funds:     $30.000.00 Q. 

2 

< 


RFP  Contact  Name:     Dr.  lohn  Lee 


Address:     Division  of  Research ____ 

School  Department,  108  Main  Street,  Cosmopolis,  MA  a 

Telephone:     (617)  123-4567  ^*j 

Due  Date:     lune  10. 1982 

Location:     Division  of  Research ^^ 

Time:     5:00  P.M. ^ 

• 

Number  of  copies  to  be  submitted: _ U^ 

(Upon  receipt,  proposals  become  the  property  of  the  District  and  will  not  be  returned  to  the  bidder.)  mtJ 


Written  Intent  to  Bid  Required?  (If  yes,  only  those  submitting  written  intent  will  be  notified  of  amendments 
and  sent  any  supplementary  materials.) 


S 


H     Yes                   D      No  ^ 

If  yes,  must  be  received  by a 

uu 


5 


s 


UJ 


UJ 


UJ 

< 


LU 

a. 

< 


The  period  of  services  will  begin  with  contract  authorization,  which  is  estimated  to  be  July  1,  1982  and 
concluded  no  later  than  August  30. 1984 


< 

(^  □     Bidder's  conference 


Further  information  about  this  RFP  may  be  obtained  in  the  following  way(s): 


Date: 
Time: 


Location: 


UJ 

CL 

^  £1     Telephone  requests  for  information 

^^  Contact  person:  Dr.  lohn  Lee 

(/)  Telephone:  (617)  123-4567,  Ext.  89 


Telephone  requests  accepted  until:  June  5, 1982 

£1     *Written  requests  for  information 

Contact  person:  Dr.  lohn  Lee 


Address:  Division  of  Research 


2 

i^f  Written  questions  must  be  received  by:  June  1, 1982 

"  H     Review  of  additional  — 

A  Location:  Same  as  above 


■U  Contact  for  appointment:   

Oh  Available  for  review  until:  

d£  El     These  supplementary  materials  will  be  sent  upon  receipt  of  written  Intent  to  Bid: 

r^  Proposal  for  the  Establishment  of  a  Pre-School  Language  Development  Program 


Proposed  Materials  and  Curriculum  for  the  Pre-School  Language  Development  Program 

Approved  Goals  and  Objectives  in  Language  Development 

*A  record  of  questions  asked  and  answers  given  will  be  kept  and  made  available  for  inspection. 
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I.  INTRODUCTION 


(A)  Overview 


This  a  formal  request  for  the  submission  of  proposals  to  assist  the  Cosmopolis  School  Department  in 
evaluating...  The  Pre-School  Language  Development  Program. 


The  scope  of  work  will  include: 


UJ 

a. 
< 


Specifically,  proposals  are  invited  to:  assist  the  Division  of  Research  in  evaluating  the  impact  of  the  program 

on  teachers  and  students  in  6  Cosmopolis  kindergarten  classes.  U^ 


2 


•  design  of  a  study  which  will  investigate  the  extent  of  implementation  of  the  project  over  a  2-year  period. 

•  implementation  of  a  full-scale  assessment  program  of  pupil  achievement  in  a  sample  of  the  6  classes       ^^ 
which  are  participants  in  the  project  ^A 

•  preparation  of  two  interim  reports  which  will  include  an  analysis  of  organizational  factors  that  affect  the 
efficiency  of  the  project 

•  preparation  of  a  final  report  which  includes  both  quantitative  and  qualiative  data  in  the  evaluation  of  the 
project 

While  it  is  expected  that  the  contractor  will  provide  a  detailed  working  schedule  (Planning  Document),  the 
following  schedule  of  critical  activities  should  be  noted  (assuming  a  contract  approval  date  of  July  1, 1982) : 

Task  Completed  by 

Detailed  planning  document  of  activities, 

procedures  and  schedule  July  30, 1982 

Interim  Report  July  30, 1983 

Final  Report  August  31, 1984 

The  above  dates  are,  of  course,  dependent  on  the  effective  authorization  date  for  beginning  the  study. 

(B)  Background 

This  section  should  provide  information  about  the  project  or  program  being  evaluated,  the  relevant 
laws  and/or  regulations,  and  previous  work  the  agency  has  done  related  to  this  project.  Providing  this 
background  helps  bidders  understand  the  purpose  and  context  for  this  evaluation.  Any  previous  results 
and/or  materials  from  previous  evaluations  should  be  cited  in  this  section. 

The  Pre-School  Language  Development  Program  (PLDP)  is  a  two-year  bilingual  educational  program  that 
has  been  funded  by  Federal  Grant  No.  K013  under  the  Bilingual  Educational  Act  93-70.  Its  purpose  is  to 
improve  the  English  speaking  and  comprehension  ability  of  children  for  whom  English  is  not  their  native 
language. 

The  program  will  take  place  in  6  Cosmopolis  elementary  schools  which  report  that  at  least  10%  of  their 
entry-level  pupils  do  not  speak  English  as  a  first  language.  It  is  open  to  all  children  residing  in  Cosmopolis 
who  1)  will  be  eligible  for  kindergarten  enrollment  in  September  1984  and  2)  do  not  speak  English  as  a  first 
language.  It  is  anticipated  that  approximately  150  pupils  will  be  eligible. 
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LU 

During  its  first  year,  the  program  will  run  from  9:00  a.m.  to  11 :30  a.m.  each  school  day  and  will  function  as  a 

nursery  school.  At  least  one  hour  a  day  will  be  devoted  to  the  Pre-School  Language  Development 

^  curriculum  which  emphasizes  role  playing,  language  games  and  phonetic  drill.  No  attempt  will  be  made  to 

^|  use  print  material.  Instead,  listening  and  speaking  skills  will  be  stressed.  The  program  is  to  run  for  2  years. 

%  After  their  initial  year  of  intensive  instruction,  pupils  will  receive  twice  weekly  two-hour  instruction 

{^  throughout  their  second  (kindergarten)  year. 


UJ 


1 

< 


UJ 


5 

< 


UJ 


During  the  1982-83  school  year,  curriculum  and  materials  for  the  program  were  developed  by  project  staff 
who  were  appointed  in  October  1981.  These  rely  heavily  upon  current  research  findings  regarding 
interpersonal  communication  skills.  This  material  will  be  used  as  the  basis  for  instruction;  however,  it  is 
anticipated  that  the  evaluator  will  provide  formative  evaluation  as  the  project  develops  and  that  the 
program  will  be  modified  accordingly. 

Full  description  of  the  development  process  as  well  as  a  set  of  instructional  materials  are  available  at  the 
offices  of  the  Division  of  Research. 

All  contractors  must  be  familiar  with  and  abide  by  those  statutes  and  regulations  dealing  with  conflict  of 
interest,  especially  regarding  state  employees  and  those  receiving  salaries  from  state  funds. 


UJ 

a. 

^^  All  contracts  will  include  all  clauses  necessary  for  concurrence  with  all  laws,  policies  and  regulations  of  the 

{f)  Commonwealth  of  Massachusetts. 


No  contractor  may  expend  project  funds  until  formal  contract  approval. 

A  contract  may  be  terminated  fifteen  working  days  after  formal  notification  to  the  contractor.  Settlement 
will  be  in  accordance  with  state  regulations. 


All  materials  produced  by  the  contractor  in  order  to  fulfill  the  requirements  of  this  proposal  shall  become 
the  property  of  the  District  and  be  submitted  to  the  Division  of  Research  prior  to  final  payment.  Included 
shall  be  printer's  negatives,  masters,  or  flats  used  in  printing  final  project  products;  computer  data  tapes; 
and  all  other  project  documents. 

If  the  contractor  proposes  to  employ  a  subcontractor(s),  the  qualifications  of  the  subcontractor(s)  shall  be 
documented  in  the  proposal  at  the  same  level  of  detail  of  those  of  the  contractor.  All  subcontractors  must  be 
mutually  agreeable  to  the  District  and  the  contractor. 
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II.  GRANT  AWARD  AND  CONTRACTUAL  INFORMATION 

(A)  Award  Criteria 

Contractor  proposals  will  be  reviewed  by  a  panel  composed  of  Division  staff.  Contractos  whose  proposals 
receive  favorable  reviews  will  be  invited  to  send  no  more  than  two  representatives  to  appear  before  the 
panel  for  a  final  proposal  evaluation.  This  evaluation  is  scheduled  for  June  20, 1982. 

Though  not  definitive,  the  award  criteria  that  the  review  panel  will  be  asked  to  apply  are  as  follows  (on  a  100 
point  scale  the  approximate  weight  of  each  criterion  is  indicated  to  the  right  of  each  statement): 

•  Clarity  and  relevance  of  objectives  15 

•  Revelance  of  proposed  activities  25 

•  Qualifications  of  staff  25 

•  Management  plan  20 

•  Budget  15 

Contractors  not  receiving  an  award  who  wish  to  discuss  their  proposals  with ^ 

following  the  award  announcement.  Feedback  on  proposal  strengths  and  weaknesses  will  be  provided  at 
that  time,  and  the  winning  proposal  may  be  examined. 

(B)  Contract  Information 

The  District  reserves  the  right  to  reject  any  and  all  proposals  or  to  eliminate  sections  of  a  contractor's 
proposal  as  it  deems  desirable. 

It  a  contract  ensues,  the  proposal  of  the  successful  bidder  will  become  a  contractual  obligation  along  with 
any  modifications  made  jointly  by  the  District  and  the  contractor.  Failure  of  the  successful  bidder  to  accept 
this  obligation  will  result  in  cancellation  of  the  award. 

Upon  award  of  the  contract,  the  contractor  will  meet  with  the  staff  of  the  District  to  discuss  detailed  plans 
and  activities.  Following  this  discussion  the  contractor  will  prepare  a  detailed  Planning  Document  for  use  by 
the  contractor,  the  District  and  others  involved  with  the  project. 

Attached  to  all  contractor  proposals  must  be  a  completed  Statement  of  Assurances,  attached  to  this  Request 
for  Proposals.  In  addition,  the  contract  awarded  the  successful  bidder  will  include  language  consistent  with 
Executive  Order  11246,  assuring  equality  of  employment  opportunity.  All  potential  contractors,  therefore, 
should  familiarize  themselves  with  this  Order.  Finally,  a  successful  bidder  who  employs  over  50  persons  will 
be  required  to  provide  the  District,  upon  request,  with  an  Affirmative  Action  Plan  consistent  with  federal 
agency  guidelines. 
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III.  PROCEDURAL  DIRECTIONS 

All  proposals  shall  include,  at  a  minimum,  the  information  shown  below  in  the  general  format  described. 
The  proposal  shall  be  divided  into  four  parts:  summary,  narrative,  budget,  appendices.  Each  is  briefly 
described  below. 

(A)  Summary 

The  summary  section  shall  include  the  proposal  title;  the  name,  title,  affiliation,  address  and  telephone  of 
the  contract  officer;  a  proposal  summary  of  500  words  or  less;  the  period  during  which  the  proposal  is  valid; 
and  the  total  budget  figure. 

(B)  Narrative 

The  narrative  portion  should  describe  the  contractor's  understanding  of  the  issues  and  problems  in 
designing  the  evaluation  project,  as  well  as  plans  for  carrying  out  the  project.  The  narrative  portion  should 
be  organized  into  technical  and  management  sections. 

The  technical  section  should  indicate  the  contractor's  understanding  of  (a)  the  issues  and  needs  involved  in 
Massachusetts;  (b)  the  overall  scope  of  the  work  as  well  as  the  specific  tasks  to  be  performed;  (c)  the 
problems  to  be  anticipated  and  possible  solutions;  and  (d)  the  persons,  agencies  and  groups  to  be  involved. 
It  should  describe  the  technical  plans  for  accomplishing  the  work  indicated  in  this  Request  and  specified  in 
the  contractor's  proposal. 

Specifically,  this  section  should: 

•  address  the  special  requirements  of  a  long-term  study  in  which  immediate  gains  are  differentiated  from 
long-term  results. 

•  show  sensitivity  to  the  measurement  and  effects  of  "implicit  instruction"  in  contrast  to  explicit  curriculum 
and  materials. 

In  requesting  specific  plans,  the  proposal  should  include: 

•  the  design  of  an  impact  study  which  will  include  the  different  types  of  analysis  necessary  to  measure 
achievement  gains 

•  the  longitudinal  effects  of  the  program  on  teachers'  attitudes  and  behavior  achievement 

•  the  interactional  effects  of  curriculum  and  instruction  style  on  pupils'  achievement 

•  the  effects  of  organizational  changes  on  teachers'  perceptions  and  effectiveness 

•  other  affective  and  cognitive  changes  in  pupils'  performance,  over  and  above  specific  basic  skills 
achievement 

The  management  section  should  include  a  schedule  for  the  completion  of  all  tasks,  progress  and  quality 
control  procedures,  the  assignment  of  staff  and  staff  time  to  specific  tasks  together  with  a  description  of  staff 
organization  and  qualifications. 

Included  in  this  section,  also,  may  be  documentation  of  the  contractor's  corporate  experience  and 
capability  to  provide  the  services  and  products  requested  on  schedule.  Corporate  qualifications  should 
briefly  describe  the  support  structure  adequate  for  this  study  as  well  as  indicate  experiences  similar  to  those 
requested  by  this  RFP. 
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(C)  Budget  gj 

The  budget  should  include  both  a  budget  by  task  and  a  line-item  budget.  The  latter  should  include  a  ^ 

breakdown  of  each  cost  item  under  salaries,  wages,  and  benefits;  consultants;  meetings  and  workshops;  jl 

staff  and  committee  travel;  materials,  supplies,  phones  and  mailing;  duplication  and  printing;  other  direct  ^^ 

costs;  indirect  costs;  and  fees.  The  basis  for  calculating  major  costs  items  should  be  explained.  All  \^ 
subcontracts  should  be  described. 

• 

(D)  Appendices  U- 

■ 

The  appendices  may  include  staff  resumes,  samples  of  previous  work,  review  of  corporate  work,  references 

and  current  telephone  numbers  of  clients  for  whom  similar  work  has  been  done,  and  any  other  materials  ■■ 

deemed  helpful  in  understanding  the  proposal  or  the  contractor's  capability  to  carry  out  the  tasks  required  V 

by  the  RFP  and  described  in  the  proposal.  ^^ 


uu 


2 

< 

uu 

< 
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UU 
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-J  IV.  TECHNICAL  SPECIFICATIONS 
EL 

5  Contractors  should  note  the  following  about  products. 

^^  •  The  contracting  unit  will  approve  and  sign-off  on  final  copy  of  all  project  materials. 

U)  •  Contractors  should  cite  specifications  and  costs  for  each  of  the  above  in  their  proposal.  Final  printing 

specifications  of  all  deliverables  must  be  approved  by  July  30, 1984. 

W  •  Contractors  have  the  responsibility  for  preparing,  duplicating,  and  mailing  or  delivering  all  products 

■  ■  .  noted  above. 

***{  •  All  products  will  be  noted  as  reports  or  publications  of  the  district  and  will  not  include  the  name  of  the 

[^  contractor.  Appropriate  acknowledgements  will  be  made  by  the  district. 

■■  •  The  format  of  the  final  reports  and  manuals  will  follow  specifications  to  be  provided  by  the  district  (i.e. 

^  names  of  advisory  committees  and  committee  members,  listing  of  previous  publications,  affirmative 

^^  action  statements,  Introduction  by  district,  etc.). 


< 


{^  1.  Assessment  Measures 


uu 

< 


s 
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a.  The  contractor  will  be  responsible  for  developing  an  Observation  Schedule  which  will  focus  on  the 
behaviors  that  are  cited  in  "Approval  Goals  and  Objectives  in  Language  Development".  This  schedule 
should  yield  quantitative  data  that  can  be  analyzed  by  multi-variate  techniques,  as  well  as  qualitative 
data  that  can  be  used  to  explain  the  effects. 

b.  The  contractor  will  be  responsible  for  developing  a  teacher  attitude  questionnaire  that  will  measure 
teachers'  endorsement  of  those  aspects  of  the  program  which  the  Division  and  the  Advisory  Committee 
deem  significant. 

c.  The  contractor  will  develop  2  forms  of  a  Mastery  test  of  the  Pre-School  Language  Development  Program 
objectives.  The  content  validity  of  these  forms  will  be  approved  by  the  Advisory  Committee. 


■U  2.  Implementation  Data 

Oh  a.     The  contractor  is  responsible  for  developing  a  schedule  for  observation  in  the  schools  on  a  weekly  basis. 

The  data  collected  will  be  analyzed  for  possible  trends  in  instructional  and  attitudinal  variables. 


i^£  b.     Pupils'  behavior  should  be  measured  for  both  direct  and  indirect  effects  of  the  program.  Specifically, 

r*L  pupils'  progress  should  be  reported  in  terms  of  achievement  in  basic  skills  and  in  cognitive 

development,  as  well  as  in  development  of  communication  skills  (i.e.,  the  Pre-School  Language 
A  Development  Mastery  tests). 


3.  Analysis 


■■■  Analysis  should  include: 

^m  a.     evidence  for  the  validity  and  reliability  of  the  instruments  developed 

i^T  b.     evidence  for  significant  differences  in  attainment  in  the  targeted  groups  versus  control. 

r^H  c.     evidence  for  significant  achievement  results  over  time. 

£  4.  Sample 

■U  During  the  first  year,  the  sample  should  contain  between  120-150  children  among  the  6  schools  that 

■J  have  been  chosen  for  the  program.  During  the  second  year,  a  school  sample  will  be  included  which  will 

be  matched  to  the  targeted  group  in  terms  of  age  and  ability  as  measured  by  the  Otis-Lennon  Ability 

Scale. 
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5.  Project  Participation 

Contractors  will  be  responsible  for  seeing  that  Division  personnel  project  leaders  and  the  Advisory 
Committee  participate  at  critical  junctures  in  the  project.  Specifically,  the  contractors  should  plan  on 
the  following: 

•  a  sufficient  number  of  meetings  with  the  Division  personnel  (3  staff)  during  the  course  of  the  study 

•  monthly  meeting  with  project  staff  to  discuss  observational  reports,  changes  in  policy,  etc. 

•  up  to  four  meetings  with  the  Advisory  Committee  whose  responsibility  it  is  to  provide  overall 
advice  on  the  study. 

Contractors  will  not  have  to  pay  honoraria  for  any  meeting  nor  provide  meeting  space.  They  will  be 
responsible,  however,  for  light  refreshments  at  all  meetings. 

6.  Printing  and  Deliverables 

The  following  are  the  major  deliverables  stemming  from  this  RFP: 


CL 

< 

UU 

EI 
< 

UU 


Product 

Planning  Document 

Report  on  Implementation  Schedule 

Monthly  Reports  to  Project  Staff 

Interim  Report 

Final  Report 


Quantity 

330 
1030 

2 
1030 
5031 


Delivery  Date 

a. 

July  1982 

2 

March  1983 

< 

June  1983 

</) 

August  1984 

* 

uu 

a. 

< 
uu 

< 
UU 
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gj  PRE-AWARD  ASSURANCES  OF  EQUAL  EMPLOYMENT  OPPORTUNITY 

^^  I  hereby  certify  that (name  of  contractor)  complies  with  all 

<  federal  and  state  statutes  that  require  equal  opportunity  in  all  aspects  of  employment  irrespective  of  race, 

color,  sex,  religion  or  national  origin.  This  includes,  but  is  not  limited  to: 
</) 
^  1)  Non-discriminatory  recruiting  and  hiring 

^y  2)  Non-discriminatory  placement  and  promotion 

3)  Non-discriminatory  pay,  other  compensation  and  working  conditions 

^  4)  Non-discriminatory  demotion,  transfer,  lay-off  and  termination 

%  5)  Non-discriminatory  subcontracting 

</) 

^  6)  Establishment  of  a  written  affirmative  action  plan  pursuant  to  Department  of  Education  guidelines, 

w  if  appropriate 

UJ 

a. 

< 

• 

UJ 

CL 

< 

UJ 

< 


0_ 

< 


Date  Signed 


Name  and  Title 
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Sample  Contract  for  a  Large  Evaluation 

This  agreement  entered  into  as  of  this  1st  day  of  July,  1982  by  and  between  the  school  district  of  Cosmopolis, 
hereinafter  referred  to  as  the  "district,"  and 

Evaluators  Anoymous,  Inc. 

95  Chi  Square 

Cambridge,  Massachusetts  02138 

Hereinafter  referred  to  as  the  "contractor". 

Whereas,  the  design  submitted  by  the  contractor  for  the  evaluation  of  the  Pre-School  Language 
Development  Program  (PLDP)  has  been  modified  by  negotiation;  and, 

Whereas,  the  School  Board  of  Cosmopolis  approves  the  recommendation  of  their  negotiation  committee  to 
proceed  with  the  evaluation;  and 

Whereas,  the  contractor  is  willing  to  undertake  this  endeavor; 

Therefore,  the  parties  do  mutually  agree  as  follows: 

1.  Period  of  Contract 

This  contract  shall  be  in  force  from  1  July,  1982  to  31  August,  1984  which  time  the  services  agreed  to  are  to 
have  been  provided  and  all  products  delivered  as  called  for  in  this  contract  and  in  accord  with  the 
deadline  included  herein. 

2.  Scope  of  Services 

The  contractor  shall  perform  those  services  necessary  to  accomplish  the  evaluation  of  the  Pre-School 
Language  Development  Program  in  accord  with  the  agreed  upon  design.  Specifically,  the  contractor 
shall: 

Observe  each  of  the  6  classes  for  a  minimum  of  4  hours  per  month.  One  half  of  this  observation  time  will 
be  devoted  to  the  use  of  the  Implementation  Schedule  which  the  contractor  will  develop.  The 
remaining  half  will  be  nonstructured.  Monthly  progress  reports  will  be  provided  to  the  project  staff. 

On  30  March,  1983  the  contractor  will  submit  a  report  on  any  modification  made  in  the  Implementation 
Observation  Schedule,  as  well  as  a  report  on  its  characteristics.  Included  in  this  report  will  be  evidence 
concerning  its  objectivity  as  well  as  a  correlational  analysis  of  the  relationship  between  teacher 
self-report  data  and  the  observation  data. 

A  final  report  on  data  collected  by  use  of  the  schedule  will  be  included  in  the  interim  report  due  on  June 
1983  and  the  final  report  due  on  31  August,  1984. 

During  the  second  year  of  the  program  (1983/84),  a  control  group  in  each  school  will  be  selected.  These 
pupils  will  be  matched  in  age  and  ability  with  the  initial  sample.  They  will  not  participate  in  the  twice 
weekly  Pre-Language  Development  Program  curriculum  but  will  be  instructed  in  the  district-wide 
pre-reading  program  which  is  in  effect  in  all  district  kindergartens. 
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During  the  months  of  September  1982  and  May  1983,  each  student  in  the  experimental  group  will  be 
administered  the  Wright   Development  Test  by  qualified  test  administrators  employed   by  the 
^  contractor.  The  test  is  to  be  individually  administered.  Students  will  receive  alternate  forms  on 

^Z  successive  administrations. 

< 

^^  The  contractor  will  be  responsible  for  all  statistical  analysis.  Performance  of  the  experimental  group  will 

be  contrasted  with  that  of  the  control  group  using  the  district-developed  test  of  basic  skills  and  the 
4P  Pre-School  Language  Development  Mastery  Tests. 


UJ 


The  contractor  will  develop  two  parallel  forms  of  a  Mastery  test  of  the  Program  objectives  for 
administration  to  the  experimental  group  on  three  occasions:  June  1983,  September  1983  and  June  1984. 
The  analysis  of  the  resulting  data  will  include  the  percent  of  students  who  have  mastered  the  various 
objectives  at  the  defined  criterion  levels.  These  data  are  also  to  be  presented  so  as  to  permit  comparison 
among  classrooms  by  implementation  indices. 


uu 

a. 

< 

U)  The  contractor  will  develop  a  teacher  questionnaire  form  which  will  include  a  section  permitting 

teachers  to  rate  themselves  in  terms  of  their  implementation  of  the  program.  The  questionnaire  is  to  be 

^  submitted  to  the  Advisory  Committee  for  review  by  September  15, 1982  and  revised  after  negotiations 

concerning  changes.  These  questionnaires  are  to  be  administered  during  November  1982  and  May 
1984.  Collection  of  this  questionnaire  data  will  be  the  responsibility  of  the  school  district.  The  contractor 

■^  will  present  frequency  and  correlational  analysis  of  the  questionnaire  data  in  the  final  evaluation  report. 

■■■  During  the  course  of  the  evaluation,  monthly  progress  reports  to  the  project  directors  are  to  be 

"^T  provided.  These  reports  are  to  be  kept  confidential. 

«^T  A  technical  report  on  the  Implementation  Observation  Schedule  is  due  in  March  1983.  Ten  copies  of 

r^  this  report  are  to  be  provided  to  the  school  district  office. 

Interim  report  of  the  evaluation  is  due  on  30  June  1983.  This  report  will  contain  an  analysis  of  the  data 
collected  to  date.  Ten  copies  will  be  provided. 

The  final  report  is  to  preserve  the  confidentiality  of  all  teachers  and  students  as  stipulated  in  this 
contract. 

3.  Cooperation  of  the  School  District 

To  facilitate  the  accomplishment  of  this  evaluation,  the  school  district  commits  itself  to  undertake  and 
accomplish  the  following: 

The  school  district  will  arrange  for  the  classes  in  the  6  schools  to  be  selected  and  functioning  by  5 
September  1982. 

The  school  district  will  provide  the  contractor  with  a  schedule  of  the  planned  classroom  use  of  the 
program  materials  within  each  participating  classroom.  This  schedule  will  be  updated  on  a  monthly 
basis  to  insure  adequate  observational  coverage. 

The  school  district  will  insure  that  the  contractor  has  access  to  the  selected  classrooms  for  observational 
purposes.  To  promote  fidelity  of  the  information  obtained,  the  timing  of  the  contractor's  observational 
visits  need  not  be  pre-arranged  with  the  teacher  nor  the  school  principal. 
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The  school  district  will  collect  and  provide  to  the  contractor  Mastery  Test  data  for  each  student  in  the 
program.  These  tests,  of  which  there  are  two,  will  be  administered  to  the  students  during  June  1983, 
September  1983  and  May  1984.  All  mastery  test  data  to  be  analyzed  will  be  provided  to  the  contractor  on 
keypunched  80  column  cards  in  a  mutually  agreed  upon  format. 

The  school  district  will  keypunch  the  Wright  Development  Test  data  on  80  column  cards  according  to  a 
mutually  agreed  upon  format.  These  cards  will  be  provided  to  the  contractor  within  15  calendar  days 
after  the  delivery  of  the  information  to  be  keypunched  to  the  school  district  offices. 

The  school  district  will  establish  an  Advisory  committee  comprised  of: 

Dr.  A.  C.  Sharpe,  the  Superintendent  of  the  Cosmopolis  School  District 

Dr.  Peter  Smith,  the  Director  of  Elementary  Instruction 

Ms.  Anne  Jones,  the  Principal  of  Elementary  Instruction 

Ms.  Paula  Roberts,  a  representative  chosen  by  the  districts'  kindergarten  teachers 

Dr.  Wendell  Beasely,  Professor  of  Educational  Psychology,  University  of  Massachusetts 

This  committee  will  review  the  technical  report,  the  teacher  questionnaires  and  the  first  draft  of  the  final 
report  and  return  their  comments  and  criticisms  to  the  contractor,  within  15  calendar  days  of  receipt. 
The  committee  will  also  serve  as  the  negotiating  body  for  reaching  agreement  concerning  necessary 
changes  in  these  documents. 

4.  Additional  Contract  Provisions 

A-  Non-Discrimination 

The  contractor  agrees  that  in  performing  this  contract  he  or  she  shall  not  discriminate  against  any 
worker,  employee  or  applicant,  or  any  member  of  the  public,  because  of  race,  sex,  creed,  color  or 
national  origin,  nor  otherwise  commit  an  unfair  labor  practice.  The  contractor  further  agrees  that  this 
clause  will  be  incorporated  in  all  contracts  entered  into  with  suppliers  of  materials  or  services, 
contractors  and  subcontractors  and  all  labor  organizations,  furnishing  skilled,  unskilled  and  craft  union 
skilled  labor,  or  who  may  perform  any  such  labor  or  services  in  connection  with  this  contract. 

B-  Compliance  With  Laws 

The  contractor  shall  at  all  times  observe  and  comply  with  all  laws,  ordinances  and  regulations  of  the 
federal,  state,  local  and  city  government,  which  may  in  any  manner  affect  the  performance  of  the 
contract. 

C.  Insurance 


The  contractor  will  purchase  and  maintain  during  the  life  of  this  contract  insurance  coverage  which  will 
satisfactorily  insure  him  against  claims  and  liabilities  which  could  arise  because  of  the  execution  of  this 
contract. 

D.  Conflict  of  Interest 

No  member  of  the  governing  body  of  the  City  of  Cosmopolis  or  other  unit  of  government  and  no  other 
officer,  employee,  or  agent  of  the  City  or  other  unit  of  governement  who  exercise  any  functions  or 
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responsibilities  in  connection  with  the  carrying  out  of  the  Project  to  which  this  contract  pertains,  shall 

have  any  personnal  interest,  direct  or  indirect,  in  this  contract.  The  contractor  convenants  that  he 

^  presently  has  no  interest  and  shall  not  acquire  any  interest  direct  or  indirect,  in  the  project  to  which  this 

^^  contract  pertains  which  would  conflict  in  any  manner  or  degree  with  the  performance  of  his  services 

^^  hereunder.  The  contractor  further  convenants  that  in  the  performance  of  this  contract  no  person 

{^  having  any  such  interest  shall  be  employed. 


UJ 


UJ 
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Confidentiality 

The  contractor  agrees  to  preserve  the  confidentiality  of  all  subjects  participating  in  this  evaluation:  no 
teacher,  student  or  school  will  be  identified  or  identifiable  in  the  written  or  oral  reports  provided  to  the 
district  or  any  other  party.  (The  use  of  fictitious  names  is  permitted  in  reports  to  illustrate  individual  uses 
if  that  is  deemed  a  desirable  reporting  technique). 


s 

^%  F.  Prior  Review 


No  written  report  of  this  evaluation  will  be  released  by  the  contractor  to  any  party  without  the 
concurrance  of  the  District  Review  Committee.  The  district  similarly  agrees  to  secure  the  approval  of 
the  contractor  prior  to  the  dissemination  of  evaluation  findings. 

G.  Other  Research  Interests 


uu 

a. 

^  The  contractor  is  free  to  use  the  instruments  developed  for  use  in  this  project  or  the  results  derived 

^^  therefrom  in  other  pursuits  as  long  as  this  use  does  not  violate  any  other  contractual  provision. 

t^  H.  Publication  Rights 


The  contractor  is  free  to  publish  a  report  of  this  evaluation  in  a  professional  journal  or  to  present  an 
account  at  a  professional  society  meeting  with  the  following  restrictions: 

1.  The  confidentiality  of  the  district,  as  well  as  that  of  the  individuals  involved,  is  preserved. 

2.  The  article  or  presentation  is  first  submitted  to  the  district  for  prior  approval  (see  section  4F). 


t^f  I.  Negotiations 

In  the  event  of  a  dispute  between  the  client  and  the  contractor  concerning  any  provision  of  this 
£  contract,  the  dispute  will  be  submitted  to  a  panel  for  arbitration.  The  panel  will  be  comprised  of: 

■U  Dr.  Michael  Roberts,  Professor  of  Education,  The  University  of  Massachusetts  Boston 

Qh  Dr.  Mary  Bilbo,  Associate  Commissioner  of  Public  Instruction,  Massachusetts  Department  of  Education 

^£  The  findings  of  this  panel  will  not  be  binding  on  either  party. 

J-  Tennlnation  of  Contract 


This  agreement  may  be  terminated  by  written  mutual  consent.  In  the  event  of  termination  the 
contractor  shall  be  reimbursed  for  its  costs  incurred  to  the  date  of  termination. 
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5.  Milestone  Dates 

The  dates  on  which  products  must  be  delivered  and/or  major  activities  must  be  concluded  are  shown 
below: 


1982/83 

1 

July 

30 

July 

15 

September 

30 

September 

September 

November 

30 

March 

15 

May 

15 

June 

30 

June 

1983/84 

10 

September 

15 

September 

May 

15 

May 

1 

June 

31 

August 

•  Contract  Agreement 

•  Planning  Document 

•  Draft  of  Implementation 
Questionnaire  delivered  for  review 

•  Classroom  Observation  Begins 

•  Administration  of  Wright  Development 
Test 

•  Administration  of  Teacher 
Questionnaire 

•  Report  on  Implementation  Schedule 

•  Administration  of  Wright  Development  Test 

•  Administration  of  the  Mastery  Test 

•  Interim  Report 


•  Selection  of  the  control  group 

•  Administration  of  the  Mastery  Test 

•  Administration  of  Teacher 
Questionnaire 

•  Administration  of  the  Wright  Development 
Test 

•  Administration  of  the  Mastery  Tests 

•  Final  Report 
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6.  Basis  of  Payment 

The  district  agrees  to  pay  the  sum  of  $30,000.00  to  the  contractor  for  the  conduct  of  this  evaluation.  Payment 
is  to  be  made  as  follow: 


1  July  1982  $10,000.00 

1  May  1983  10,000.00 

30  August  1984  10,000.00 

The  contractor  assures  the  district  that  the  indirect  costs  charged  against  this  contract  will  not  exceed  20.5% 
of  the  direct  costs  incurred.  In  the  event  of  contract  termination,  the  contractor  will  be  reimbursed  for  its 
costs  to  the  date  of  termination. 


UJ  School  District  of  Cosmopolis  Evaluator's  Anonymous,  Inc. 

Qh  By By 

_C  Date Date 

< 


a. 
< 
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III.     CONDUCTING  A  SMALL  EXTERNAL  EVALUATION 

For  purposes  of  our  discussion,  a  small  external  evaluation  is  one  less  than  $10,000.  In  these  cases  a  district 
may  choose  to  not  go  through  the  detailed  processes  described  in  Section  II  of  this  guide.  Many  of  the  same 
steps  would  still  be  followed,  however,  and  we  will  refer  to  earlier  sections  where  appropriate. 

Selecting  the  Evaluator 

This  section  has  been  divided  into  two  cases:  those  evaluations  $2,000  to  $10,000,  and  those  evaluations  less 
than  $2,000.  School  districts  need  not  go  out  for  bid  on  contracts  less  than  $2,000  according  to  Chapter  40 
Section  4B  of  the  Massachusetts  General  Laws. 

Evaluations  $2,000  -  $10,000 

A  district  will  probably  issue  an  RFP  for  this  evaluation,  but  the  process  may  not  be  as  formal  or  detailed  as 
the  one  described  in  Section  II  of  this  report.  A  district  can  scale  down  the  activities  as  it  sees  fit. 

It  is  possible  that  for  this  size  contract  an  overall  evaluation  design  has  already  been  decided,  perhaps 
required  by  program  regulation  or  law  or  simply  determined  by  the  district.  An  evaluation  design  is  a 
complete  plan  of  the  work  required,  i.e.,  data  to  be  collected,  groups  to  be  contacted,  money  available  for 
the  evaluation. 

Two  sample  documents  are  included  at  the  end  of  this  section:  a  sample  RFP  and  a  sample  contract.  The 
RFP  is  an  informal  version  of  the  typical  RFP.  The  major  point  of  sending  it  is  to  ascertain  contractors  willing 
to  do  the  work.  The  district  is  then  likely  to  choose  an  evaluator  amongst  those  willing,  on  the  basis  of  the 
training  and  experience  of  the  consulting  staff  who  would  be  conducting  the  evaluation.  This  is  a  suggested 
approach  although  a  district  could  again  be  as  formal  or  informal  in  selecting  the  evaluator  as  it  chooses. 

The  sample  contract  also  is  a  simpler  version  of  the  contract  in  Section  II  of  this  report.  It  includes  the 
necessary  elements  of  a  good  working  contract  and  is  useful  for  any  contract  $10,000  or  less.  (The  example 
happens  to  be  for  one  for  $3,000). 

Contracts  less  than  $2,000 

For  proposals  less  than  $2,000  a  school  district  need  not  go  out  for  bid  and  issue  an  RFP.  Chapter  40  Section 
4B  of  the  General  Laws,  allows  them  this  discretion.  The  school  district  could  issue  an  RFP  but  would 
expect  a  very  small  response  from  contractors  unless  a  special  effort  was  made  to  find  individual  evaluators 
in  need  of  work.  It  "costs"  contractors  time  and  money  to  respond  to  an  RFP  (typing,  writing,  mailing  costs) 
and  would  in  general  not  find  it  worth  their  while  to  respond  to  such  a  small  contract. 

In  this  case,  a  district  might  directly  contact  an  evaluator  to  offer  the  contractor  the  work.  Assuming  the 
evaluator  is  free  to  do  the  work,  after  is  has  been  explained  to  the  contractor,  the  school  district  and  the 
contractor  should  meet  to  discuss  the  evaluation  and  work  out  a  contract. 

While  much  of  the  initial  communication  between  the  district  and  contractor  is  in  the  form  of  a  "gentlemen's 
agreement,"  it  is  still  important  for  the  district  to  have  worked  out  in  writing  just  what  the  evaluation  will 
entail.  The  questions  listed  in  this  report  for  writing  an  RFP  are  helpful  here  (Refer  to  page  2).  This 
information  is  helpful  to  the  evaluator  so  that  both  parties  have  the  same  expectations  about  the  content  of 
the  evaluation.  It  is  also  useful  in  writing  the  contract. 

A  major  consideration  in  hiring  a  contractor  this  way  is  how  to  maintain  the  credibility  and  the  objectivity  of 
the  evaluation  report  so  there  is  no  apparent  conflict  of  interest.  A  school  district  should  have  the  utmost 
confidence  in  the  work  of  the  evaluator  through  previous  direct  work  completed  for  the  district  or  from 
references  the  school  district  has  recently  contacted. 

Monitoring  and  Evaluating  the  Evaluation  Contract 

Monitoring 

Monitoring  an  evaluation  contract  is  an  essential  part  of  the  evaluation  manager's  job.  For  a  discussion  of 
monitoring  refer  to  page  9  of  this  report  which  describes  monitoring  a  large  evaluation  contract.  All  of  the 
discussion  there  is  pertinent  to  monitoring  a  small  evaluation  contract.  Even  if  there  is  no  formal  RFP  or 
formal  Contract  Management  Plan,  a  district  should  consider  writing  less  formal  documents,  in  the  form  of 
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memos  between  the  contractor  and  district,  to  cover  these  topics.  It  is  still  important  to  have  expectations 
mutually  agreed  upon  and  in  writing. 

Evaluating 

A  school  district  should  have  a  plan  to  evaluate  the  quality  of  contract  work.  For  a  discusssion  of  evaluation 
see  page  9  of  this  report,  which  describes  evaluation  of  a  large  contract.  While  the  discussion  is  relevant 
here,  with  the  small  evaluation  the  feedback  to  evaluators  may  be  informal  (verbal)  rather  than  formal. 
Feedback  is  important  but  the  district  will  have  to  weigh  the  administrative  costs  involved  in  formalizing  this 
feedback. 

Non-Compliance  Procedures 

See  page  12  of  this  report  for  this  discussion. 
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Sample  "RFP"  for  a  Small  Evaluation 


UJ 


< 


Memorandum  to: 


TheCosmopolis  PublicSchools  will  require  the  services  of  an  outside  evaluator  for  a  newTitle  VII  Bilingual  ^j 
Education  Project.  The  school  system  is  notifying  a  number  of  possible  consultants  and  soliciting  their 
interests  in  providing  this  service.  This  form  may  be  completed  and  returned  to  indicated  your  interest  and 
to  indicate  the  amount  of  funds  which  you  are  requesting  for  this  service.  The  evaluation  design  within  the 

project  has  been  approved  and  is  subject  to  change  and  improvement  by  mutual  agreement  between  the  ^^ 

project  director  and  the  outside  evaluation  consultant.  The  highlights  of  the  services  of  the  outside  ^ 

evaluator  are  attached.  ^^» 

</) 
Yes,  we  are  interested  in  providing  the  service.  ^ 

No,  we  are  not  interested. 


For  this  service,  we  would  require  $ 

The  resumes  giving  the  training  and  experience  of  our  staff  who  would  be  assigned  to  this  project,        ^^ 
are  attached.  ^^ 

Signed f^r^ 

(Authorized  Representative  of  Firm) 

UJ 


(Street  Address) 


S 


(City,  State,  Zip)  ^£ 

PLEASE  RETURN  BY  APRIL  30, 1982 

< 

LU 

Q. 
< 
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Sample  Contract  for  a  Small  Evaluation 

2.  Sample  Contract  for  a  Small  Evaluation 

AGREEMENT 

This  agreement  is  made  the  first  of  July,  1982,  and  entered  into  by  and  between  the  School  District  of 
Cosmopolis,  Massachusetts,  hereinafter  called  the  district,  and  Evaluators  Anonymous,  Inc.,  hereinafter 
called  the  contractor. 

1.0  Purpose 

The  district  requires  the  services  of  the  contractor  as  authorized  and  budgeted  in  the  approved  Early 
Childhood  Act  P.L.  99-999,  grant  No.  ABC123.  The  district  and  the  contractor  mutually  agree  to  carry  out 
activities  below  that  will  result  in  the  evaluation  of  the  Pre-School  Language  Development  Program 
(PLDP). 

2.0  The  contractor  will: 


LU  2.1  Meet  with  project  staff  to  review  and  possibly  modify  the  evaluation  design. 

■■J  2.2  Develop  assessment  instruments,  questionnaires,  etc. 

Qua  2.3  Visit  project  schools  monthly  and  meet  with  project  staff. 

^^  2.4  Collect,  tabulate  and  analyze  data  with  assistance  from  project  director. 

^^  2.5  Prepare  an  interim  report  by  February  15, 1983,  and  a  final  evaluation  report  by  July  30, 1983. 

< 

t/5 


3i)  The  district  will: 


3.1  Administer  the  Gates-McGinitie  Reading  Test,  the  Prueba  deLectura,  and  Test  deDistrezas  Basics  to 
all  their  and  Fourth  Grade  students. 
UU  3.2  Provide  the  evaluator  with  a  list  of  students  who  meet  the  criteria  and  with  the  names  and  addresses 

bbJ  of  these  students'  parents  or  guardians. 

g^  3.3  Arrange  for  the  equivalent  of  one  day  per  week  of  the  Home/School  Liaison  Worker's  time  to  be 

^^  released  to  the  project. 

^£_  3.4  Cooperate  with  the  evaluator  in  arranging  for  the  collection  of  the  necessary  data. 

^^  3.5  Meet  and  cooperate  with  the  evaluator  for  project  review  and  the  improvement  of  project 

^  activities. 

3.6  Provide  the  sum  of  $3,000  to  the  contractor  for  the  conduct  of  the  evaluation. 


4.0  Conditions 

4.1  The  period  of  performance  shall  be  from  July  1, 1982  to  July  30, 1983. 

4.2  This  shall  be  fixed  price  contract  in  the  amount  of  $3,000.00  for  the  performance  of  work  stipulated 
in  paragraphs  2.1  through  2.5. 

4.3  Three  copies  of  the  final  report  shall  be  delivered  to  the  district.  All  materials  issued  from  this 
evaluation  will  remain  the  property  of  the  district. 

4.4  The  payment  schedule  to  the  contractor  will  be  as  follows: 

November  1, 1982 

(Conclusion  of  5  work  days)  $1000.00 

February  15, 1983 

(Conclusion  of  8  work  days)  $1000.00 

Acceptance  of  Final  Report  on 

or  about  July  30, 1983  $1000.00 
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4.5  This  agreement  may  be  terminated  by  written  mutual  consent.  In  the  event  of  termination  the 

contractor  shall  be  reimbursed  for  its  cost  incurred  to  the  date  of  termination.  ^^ 

School  District  of  Cosmopolis  Evaluators  Anonymous,  Inc.  ^^ 


By    By 


^ 


Date  Date 


Approved  to  appropriation:  $3,000.00  5-912-8019 


~    LU 

a. 

< 

LU 

Ob 

< 

• 
LU 

< 
LU 

a. 
< 

LU 

a. 


Highlights  of  Evaluation  Design 

The  evaluation  consultant  to  be  selected  for  this  project  must  have  at  least  one  evaluator  who  is  a  bilingual  ^^ 

person  who  is  fluent  in  Spanish  and  English.  The  evaluation  firm  should  have  previous  experience  with  g* 

bilingual  projects  or  other  type  of  similar  activities.  ^% 

The  consultant  will  provide  the  following  specific  tasks:  ^ 

1.  Meet  with  project  staff  to  review  and  possibly  modify  the  evaluation  design  (2  man  days)  ^y 

2.  Develop  assessment  instruments,  questionnaires,  etc.,  (2  man  days) 

3.  Visit  project  schools  periodically  (5  man  days)  ^^ 

4.  Collect,  tabulate,  and  analyze  data  with  assistance  from  the  project  director  (4  man  days)  % 

5.  Prepare  an  interim  report  by  February  15, 1983,  and  a  final  evaluation  report  by  July  30, 1983  (8  man  days)  ^ 

LU 

The  project  evaluation  consultant  should  work  in  cooperation  with  the  project  staff  to  see  that  project 
services  are  being  delivered,  to  assist  in  improving  the  quality  of  project  activities,  and  to  suggest 
improvements  and  modifications  in  the  project  during  this  year  or  during  the  next  year. 

The  evaluation  consultant  selected  this  year  will  continue  for  a  second  year  upon  satisfactory  completion  of  ^^ 

all  tasks  since  the  project  is  tentatively  approved  for  a  two-year  period  of  time  by  the  U.S.  Department  of  "^ 

Education.  The  specific  performance  objectives  from  the  evaluation  design  are  attached  for  your  {^ 

information.  ^ 

At  this  time,  $3,000  is  budgeted  for  the  outside  evaluator,  but  this  is  subject  to  amendment.  , ,  , 


2 
< 


LU 


2 
< 


LU 
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PERFORMANCE  OBJECTIVES 

AND 
RELATED  EVALUATION  PLANS 


Performance  Objective  #  1a. 


Target  Population: 

90%  of  the  limited  English  proficient  students  identified  by  the 
language  assessment  battery  (English)  to  be  performing  at  or 
below  the  third  stanine 

Desired  Behavior: 

will  increase  their  English  comprehenion  and  vocabulary  scores 

Performance  Level: 

by  demonstrating  one  month  gain  for  each  month  in 
the  program 

Operational  Condition: 

provided  they  receive  supplementary  instruction  at  least  150 
school  days 

Instrument: 

Gates  McGinitie  Test 

Evaluation  Plan  for  above  Objective 


UJ     General  Technique: 


group  administered  standardized  test 


QH  Baseline  Data: 

^^  Sample  Group(s): 

< 

*^*  Collection  Dates: 


October 


randomly  selected  population  (N  =  10%  of  total  bilingual 
program 


October  and  June 


UJ 


Data  Analysis: 


pre-post  analysis 


< 


UJ 

< 
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PERFORMANCE  OBJECTIVES 

AND 
RELATED  EVALUATION  PLANS 
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Performance  Objective  #  1b. 


Target  Population: 

I 

90%  of  the  limited  English  proficient  students  identified  by  the 
language  assessment  battery  (Spanish)  to  be  performing  at  or 
below  the  third  stanine 

Desired  Behavior: 

will  increase  their  acquisition  of  concepts  and  skills  through 
their  native  language 

Performance  Level: 

by  a  statistically  significant  increase  in  raw  scores 

Operational  Condition: 

provided  they  receive  supplementary  instruction  at  least  150 
school  days 

Instrument: 

Prueba  de  Lectura  and  Test  de  Distrezas  Basicas 

Evaluation  Plan  for  above  Objective 


General  Technique: 

group  administered  standardized  test 

Baseline  Data: 

October 

Sample  Group(s): 

randomly  selected  population  (N  =  10%  of  total  TBE  population) 

Collection  Dates: 

October  and  June 

Data  Analysis: 

pre-post  analysis 

UJ 

fiL 

< 

LU 

< 
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PERFORMANCE  OBJECTIVES 

AND 
RELATED  EVALUATION  PLANS 


< 


UJ 
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Performance  Objective  #  2a. 

Target  Population: 

i 

j    the  efforts  of  the  Home/School  Liaison  Worker 

Desired  Behavior: 

\    will  be  proven  effective  by  their  ability  to  have  parents  come 

into  school 

1 

Performance  Level: 

j 

i    90%  of  the  time 

i 

Operational  Condition: 

providing  they  are  informed  and  assisted  by  the  Home/School 

Liaison  Worker 

l 

Instrument: 

,_ _,...j 

as  measured  by  an  analysis  of  logs 

i 

Evaluation  Plan  for  above  Objective 

General  Technique: 

Log  summary  sheets 

Baseline  Data: 

September  and  subsequent  weeks 

Sample  Group(s): 

parents  contacted  for  that  purpose 

Collection  Dates: 

weekly 

Data  Analysis: 

monthly  analysis  of  data 

UJ 

< 


UJ 


5 
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PERFORMANCE  OBJECTIVES 

AND 
RELATED  EVALUATION  PLANS 


Performance  Objective  #  2b. 

Target  Population: 

the  efforts  of  the  Home/School  Liaison  Worker 

Desired  Behavior: 

will  be  rated  as  effective  by  the  Title  VII  principals 

Performance  Level: 

to  the  extent  that  principals  indicate  a  score  of  3  or 
better  on  an  effectiveness  scale  from  5-1 

Operational  Condition: 

providing  the  survey  is  confidential 

Instrument:                                             j    by  analysis  of  items  on  questionnaire 

1 

Evaluation  Plan  for  above  Objective 

General  Technique: 

Questionnaire 

Baseline  Data: 

N/A 

Sample  Group(s): 

all  Title  VII  principals 

Collection  Dates: 

Spring 

1 
Data  Analysis:                                            Descriptive  statistics 

2 

< 
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Appendix  A 

Program  Evaluation  Standards 

The  following  standards  were  developed  in  Spring  1982  by  the  Department  of  Education  Program  Evaluation 
Committee,  a  group  of  staff  respresenting  all  major  units  in  the  Department.  These  evaluation  standards  provide 
general  principals  for  the  conduct  and  use  of  evaluation  studies  completed  by  Department  staff  or  monitored  by 
Department  staff.  The  focus  is  on  determining  the  process,  purpose  and  scope  of  evaluations,  and  on  reporting 
evaluation  results.  The  standards  are  general  goals  for  evaluation  and  cover  all  types  of  evaluation  studies,  small  to 
large,  internal  and  external,  formative  and  summative.  Because  of  limited  resources  of  programmatic 
requirements,  however,  not  all  standards  would  apply  to  a  particular  evaluation. 

This  material  is  adapted  from  and  relies  heavily  on  materials  developed  nationally  by  the  Joint  Committee  on 
Standards  for  Educational  Evaluation  published  in  Standards  for  Evaluations  of  Educational  Programs,  Projects 
and  Materials. 

Utility  Standards 

Utility  standards  are  intended  to  ensure  that  an  evaluation  will  serve  the  practical  information  needs  of  given 
audiences. 

Audience  Identification 

The  audiences  involved  in  or  affected  by  the  evaluation  should  be  identified  so  that  their  needs  can  be 

addressed. 

•  Key  audiences  who  will  use  or  be  affected  by  the  evaluation  should  be  identified  prior  to  the  evaluation. 

•  The  evaluation  audience  should  not  be  too  general  or  diverse. 

Evaluator  Credibility 

The  persons  conducting  the  evaluation  should  be  trustworthy  and  competent  to  perform  the  evaluation,  so 

that  findings  achieve  maximum  credibility  and  acceptance. 

•  The  evaluator  should  exhibit  professional  and  technical  competence  demonstrated  through  training, 
experience,  integrity  and  public  relations  skills. 

•  The  client  should  determine  that  the  evaluation  plan  is  realistic  and  technically  sound.  A  review  panel  may 
be  used  for  this  purpose.  The  review  panel  may  consist  of  program  or  project  administrators,  teachers, 
technical  experts,  etc. 

Information  Scope  and  Selection 

Information  collected  should  be  of  sufficient  scope  and  selected  appropriately  to  address  pertinent  questions 

about  the  object  of  the  evaluation  and  be  responsive  to  the  needs  and  interests  of  specified  audiences. 

•  Questions  to  be  addressed  should  be  relevant  to  the  evaluation  purposes,  clearly  specified,  and  agreed 
upon  by  the  client,  evaluator,  and  as  appropriate,  other  affected  audiences. 

•  Evaluation  plans  should  include  a  mechanism  for  providing  periodic  progress  reports.  This  may  take  the 
form  of  a  review  panel. 

Interpretation  of  Findings 

There  should  be  prior  agreement  between  appropriate  parties  on  the  perspective,  procedures,  and  rationale 

used  to  interpret  the  findings,  so  that  the  bases  for  judgments  are  clear. 

•  Evaluation  approaches  should  be  clearly  defined  and  justified. 

•  Evaluation  plans  should  reasonably  allow  for  additional  or  unexpected  variables. 
Evaluation  plans  should  specify  individuals  or  groups  who  will  collect  and  interpret  findings,  i.e.,  teachers 
will  administer  tests,  scoring  service  will  score  tests,  data  will  be  analyzed  by  contractor,  etc. 


e 


Report  Clarity 

The  evaluation  report  should  describe  the  program/project  being  evaluated  and  its  context,  purposes, 

procedures  and  findings  of  the  evaluation,  so  that  the  specified  audiences  will  readily  understand  what  was 
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done,  why  it  was  done,  what  information  was  obtained,  what  conclusions  were  drawn,  and  what 
recommendations  were  made. 

•  Evaluation  reports  should  include  a  concise  summary  that  clearly  explains  the  purpose,  methodology, 
limitations,  findings  and  recommendations. 

•  Evaluation  reports  should  address  the  key  evaluation  questions,  findings,  and  study  limitations. 

Report  Dissemination 

Timely  evaluation  findings  that  encourage  use  and  follow-through  should  be  provided  to  clients  and  other  key 

audiences. 

•  Evaluation  plans  should  include  time  lines  for  all  reports,  including  progress  reports,  for  maximum 
audience  usage. 

•  Evaluation  plans  should  allow  for  essential  revisions  of  the  time  lines  without  sacrificing  technical  accuracy 
or  report  deadlines. 

•  Evaluation  should  include  suggestions  for  follow-through  by  audiences  who  will  assess  the  findings  and 
implement  the  recommendations. 

•  Evaluation  reports  should  be  presented  to  and  reviewed  by  the  client  prior  to  dissemination. 

•  Evaluation  summaries  with  recommendations  should  be  provided  to  all  specified  audiences  so  that  they 
can  assess  and  use  findings. 

Feasibility  Standards 

Feasibility  standards  are  intended  to  ensure  that  an  evaluation  plan  is  practical,  politically  viable  and  cost  effective 
for  the  program/project  setting. 

Practical  Procedures 

The  evaluation  procedures  should  be  practical,  so  that  disruption  is  kept  to  a  minimum  and  that  necessary 

information  can  be  obtained. 

•  The  plan  for  data  collection  instruments  and  procedures  should  be  reviewed  by  people  who  have 
experience  in  collecting  data  and  by  those  administrators,  teachers,  or  others  who  would  be  providing 
data. 

•  Determine  if  any  data  needed  for  study  is  already  available  so  that  new  data  collection  efforts  will  not  be 
redundant. 

•  Data  collection  should  be  coordinated  with  other  comparable  data  collection  activities  occurring  within  the 
same  time  period. 

•  Overall  data  collection  efforts  for  the  school  year  should  be  scheduled  so  as  to  minimize  information 
requests. 

Political  Viability 

The  evaluation  should  be  planned  and  conducted  with  the  participation  and  cooperation  of  various  interest 

groups. 

•  Representatives  from  key  constituent  groups  could  be  consulted  initially  and  periodically  to  provide 
suggestions  and  react  to  certain  aspects  of  the  evaluation  such  as,  determining  design,  selecting  a 
contractor,  reviewing  instruments,  analyzing  results,  and  planning  dissemination  strategies. 

•  Periodic  information  on  the  progress  of  the  evaluation  study  should  be  provided  to  key  constituent  groups. 

Cost  Effectiveness 

The  evaluation  should  produce  information  of  sufficient  value  to  justify  the  resources  expended. 

•  Analyze  the  financial  and  human  costs  against  the  benefits  to  be  derived  and  select  the  best  combination  of 
costs  and  benefits. 

•  Evaluations  should  be  conducted  economically. 
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Propriety  Standards 

Propriety  standards  are  intended  to  ensure  that  an  evaluation  will  be  conducted  legally,  ethically  and  with  due 
regard  for  the  welfare  of  those  involved  in  the  evaluation  and  affected  by  the  results. 

Formal  Obligation 

The  obligations  of  the  formal  parties  to  an  evaluation  (what  is  to  be  done,  for  what  purpose,  how,  by  whom  and 

when)  should  be  agreed  to  in  writing. 

•  Be  sure  all  major  components  are  part  of  a  written  agreement,  i.e.,  the  questions  to  be  investigated,  data 
analysis,  reporting  plan,  etc. 

•  Ensure  that  the  written  agreement  conforms  to  federal,  state  and  program  requirements  for  expenditures, 
privacy,  evaluation  and  reporting. 

•  Negotiate  amendments  to  the  written  agreement  as  the  work  proceeds  if  it  is  necessary  to  change  the 
scope  of  the  work,  cost,  and/or  timetable. 

Full  and  Frank  Disclosure 

Oral  and  written  evaluations  should  be  open,  direct  and  honest  in  their  disclosure  of  pertinent  findings, 

including  the  limitations  of  the  evaluations. 

Public's  Right  to  Know 

The  formal  parties  to  an  evaluation  should  respect  and  assure  the  public's  right  to  know,  within  the  limits  of 
other  related  principles  and  statutes,  such  as  those  dealing  with  public  safety  and  right  to  privacy.  The 
Department  will  make  available  legal  information  relating  to  the  public's  right  to  know,  privacy,  civil  and 
human  rights,  public  safety. 

Rights  of  Human  Subjects 

The  evaluation  should  be  designed  and  conducted  so  that  the  rights,  welfare  and  dignity  of  human  subjects  are 

respected  and  protected. 

•  The  Department  will  make  available  state  and  federal  laws  on  the  rights  of  human  subjects,  such  as  those 
related  to  participation,  the  privilege  of  withdrawing,  health  and  safety  protection,  and  privacy  of 
information. 

•  Be  sure  that  necessary  written  permissions  are  obtained,  and  guarantees  for  anonymity  to  individuals  are 
assured. 

Conflict  of  Interest 

Conflict  of  interest  should  be  approached  openly  and  honestly  so  it  does  not  compromise  the  evaluation 

process  and/or  results. 

•  In  initial  discussions  with  potential  evaluators,  identify  and  describe  possible  sources  of  conflict  of  interest. 

•  Consider  setting  a  maximum  number  of  consecutive  years  for  which  an  evaluator  may  evaluate  the  same 
program  or  project. 

Balanced  Reporting 

The  evaluation  should  be  complete  and  fair  in  its  presentation  of  strengths  and  weaknesses  of  the  program  or 

project  under  investigation,  so  that  strengths  can  be  built  upon  and  problem  areas  addressed. 

•  For  ongoing  or  recurrent  programs,  the  new  proposal  should  reflect  the  findings  of  previous  program 
evaluations. 

Fiscal  Responsiblity 

The  evaluator's  allocation  and  expenditure  of  resources  should  reflect  sound  accountability  procedures  and 

otherwise  be  prudent  and  ethically  responsible. 

•  Contractors  should  maintain  accurate  records  of  sources  of  funding  and  expenditures  in  a  clear  and 
recognized  format. 

•  A  well  defined  budget  should  be  included  in  the  formal  contract  agreement. 
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•  The  Department  will  make  available  laws  and  rules  on  approved  state  and  federal  accounting  procedures 
which  must  be  followed  whenever  evaluations  are  contracted. 

Accuracy  Standards 

Accuracy  standards  are  intended  to  ensure  that  an  evaluation  is  truthful  or  valid  in  what  it  says  about  a  program, 
project,  or  materials. 

Identifying  the  Program,  Project,  or  Materials 

The  program,  project,  or  materials  of  the  evaluation  should  be  clearly  identified  and  realistically  examined  in 

terms  of  personnel,  cost,  procedures,  location,  facilities,  goals,  objectives,  potential  side  effects,  etc. 

Context  Analysis 

The  evaluation  should  describe  the  contextual  influences  on  the  program,  project,  or  materials  including  the 
geographic  location  of  the  program  or  project,  timing,  political  and  social  climate,  the  nature  of  the  staff,  and 
pertinent  economic  conditions. 

Described  Purposes  and  Procedures 

The  purposes  and  procedures  of  the  evaluation  should  be  described  in  sufficient  detail  so  that  they  can  be 
identified  and  monitored  by  both  the  evaluator  and  the  client  to  ensure  that  the  intended  purposes  of  the 
evaluation  are  being  met. 

•  The  evaluation  report  should  describe  a  study's  objectives,  intended  uses,  and  ways  in  which  information 
was  gathered,  organized  and  analyzed. 

Defensible  Information  Sources 

The  sources  of  information  should  be  described  so  that  the  accuracy  of  the  information  can  be  assessed. 

•  The  evaluation  report  should  describe  the  sources  of  information,  including  the  criteria  and  methods  used 
to  select  them,  while  still  preserving  the  confidentiality  of  the  sources. 

Valid  and  Reliable  Measurement 

The  information-gathering  instruments  and  procedures  should  be  chosen  or  developed  and  implemented  in  a 
manner  that  will  assure  that  the  interpretation  is  valid  and  the  information  obtained  is  reliable  for  its  intended 
use.  A  valid  instrument  is  one  that  is  being  used  and  interpreted  in  an  appropriate  way.  A  reliable  instrument 
has  proven  it  consistently  measures  the  characteristics  being  investigated. 

•  Evaluation  reports  should  clearly  describe  the  procedures  used  to  ensure  valid  and  reliable  measurement 
procedures. 

•  Evaluation  reports  should  state  the  data  collected  and  the  results  of  the  analyses  performed  to  determine 
the  validity  and  reliability  of  measurement  instrument  developed  for  a  study. 

Systematic  Data  Control 

The  data  collected,  processed  and  reported  in  an  evaluation  should  be  reviewed  and  quality  control  measures 

implemented  so  that  the  results  of  evaluation  will  not  be  flawed. 

•  In  conducting  an  evaluation  study,  steps  should  be  taken  to  ensure  that  all  data  used  will  be  as  error-free  as 
is  humanly  possible.  A  systematic  program  of  training,  controls,  and  accuracy  checks  should  be  instituted. 

Quantitative  and  Qualitative  Analysis 

Quantitative  and  qualitative  information  in  an  evaluation  should  be  appropriately  and  systematically  analyzed 

to  ensure  supportable  interpretation. 

•  An  analytic  procedure  should  be  selected  that  is  appropriate  to  the  questions  addressed  in  the  study.  For 
example,  the  results  of  certain  data  analysis  should  be  tested  to  insure  that  those  results  are  statistically 
significant  events  rather  than  chance  occurrences. 

•  The  evaluation  report  should  indicate  potential  weaknesses  in  the  data  collection  or  analysis  and  describe 
their  possible  influence  on  conclusions. 
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Justified  Conclusions 

The  conclusions  of  an  evaluation  should  be  explicitly  justified  so  that  the  audience  can  assess  them. 

•  The  conclusions  in  an  evaluation  report  must  be  defensible,  i.e.,  based  on  sound  logic  and  appropriate 
information. 

•  The  conclusions  in  an  evaluation  report  must  be  defended,  i.e.,  reported  along  with  an  account  of  the 
procedures,  information  and  underlying  assumptions  and  with  a  discussion  of  possible  alternative  findings 
and  why  they  were  rejected. 

Objective  Reporting 

The  evaluation  procedures  should  provide  safeguards  to  protect  the  evaluation  findings  and  reports  against 
distortion  by  the  personal  feelings  and  biases  of  any  party  to  the  evaluation.  Evaluation  reports  should  be 
based  on  facts  selected  through  impartial  methods  and  not  slanted  to  promote  biased  positions. 
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